2023-09-30T00:15:25.877 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-09-30T00:15:25.877 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719 --verbose --owner scheduled_yuriw@teuthology --name yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi --description rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete} -- /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/orig.config.yaml 2023-09-30T00:15:25.920 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:15:26.007 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719 branch: wip-yuri4-testing-2023-09-29-0928-quincy description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete} email: yweinste@redhat.com first_in_suite: false job_id: '7406719' kernel: kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-09-29-0928-quincy ceph: conf: client: debug monc: 20 debug ms: 1 global: mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr1: false ms bind msgr2: true ms inject delay max: 1 ms inject delay probability: 0.005 ms inject delay type: mon ms inject internal delays: 0.002 ms inject socket failures: 2500 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - slow request - overall HEALTH_ - \(POOL_APP_NOT_ENABLED\) sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-09-29-0928-quincy sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 9810 sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-09-29-0928-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 targets: smithi006.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnRvJR0L2T9BYkKpGQNJp0fSnvm/Iup0o098MrH+5RqggRpFysYNoYtSqH26RVshsFZM5mKCGbBP/pXIRjBSlo= smithi022.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIS9surE7teWMyjraGAsw5v/FL+wIMOQobe4oi+WNkyXfGPxNue/tZudQy9axQWyCS951ZOY870XlEUUAonQhwo= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 thrash_many: true - exec: client.0: - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel - ceph_test_rados_delete_pools_parallel teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-09-29_19:38:56 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-09-30T00:15:26.008 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa; will attempt to use it 2023-09-30T00:15:26.008 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks 2023-09-30T00:15:26.008 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-09-30T00:15:26.010 INFO:teuthology.task.internal:Checking packages... 2023-09-30T00:15:26.033 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '93dce5e17918f8aca65993a7a2dc40233a47b481' 2023-09-30T00:15:26.033 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-09-30T00:15:26.033 INFO:teuthology.packaging:ref: None 2023-09-30T00:15:26.033 INFO:teuthology.packaging:tag: None 2023-09-30T00:15:26.034 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:15:26.034 INFO:teuthology.packaging:sha1: 93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:15:26.034 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-09-29-0928-quincy 2023-09-30T00:15:26.192 INFO:teuthology.task.internal:Found packages for ceph version 17.2.6-1165.g93dce5e1 2023-09-30T00:15:26.192 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-09-30T00:15:26.199 INFO:teuthology.task.internal:no buildpackages task found 2023-09-30T00:15:26.199 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-09-30T00:15:26.215 INFO:teuthology.task.internal:Saving configuration 2023-09-30T00:15:26.231 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-09-30T00:15:26.237 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:15:26.263 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:09:13.969016', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnRvJR0L2T9BYkKpGQNJp0fSnvm/Iup0o098MrH+5RqggRpFysYNoYtSqH26RVshsFZM5mKCGbBP/pXIRjBSlo='} 2023-09-30T00:15:26.287 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:09:13.967196', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIS9surE7teWMyjraGAsw5v/FL+wIMOQobe4oi+WNkyXfGPxNue/tZudQy9axQWyCS951ZOY870XlEUUAonQhwo='} 2023-09-30T00:15:26.288 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-09-30T00:15:26.294 INFO:teuthology.task.internal:roles: ubuntu@smithi006.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2023-09-30T00:15:26.294 INFO:teuthology.task.internal:roles: ubuntu@smithi022.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2023-09-30T00:15:26.294 INFO:teuthology.run_tasks:Running task console_log... 2023-09-30T00:15:26.368 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f23bbdc7ee0>, signals=[15]) 2023-09-30T00:15:26.369 INFO:teuthology.run_tasks:Running task internal.connect... 2023-09-30T00:15:26.375 INFO:teuthology.task.internal:Opening connections... 2023-09-30T00:15:26.375 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:15:26.377 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:15:26.445 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:15:26.446 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:15:26.515 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-09-30T00:15:26.521 DEBUG:teuthology.orchestra.run.smithi006:> uname -m 2023-09-30T00:15:26.545 INFO:teuthology.orchestra.run.smithi006.stdout:x86_64 2023-09-30T00:15:26.546 DEBUG:teuthology.orchestra.run.smithi006:> cat /etc/os-release 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:NAME="CentOS Stream" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION="8" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:ID="centos" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:VERSION_ID="8" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:15:26.608 INFO:teuthology.orchestra.run.smithi006.stdout:ANSI_COLOR="0;31" 2023-09-30T00:15:26.609 INFO:teuthology.orchestra.run.smithi006.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:15:26.609 INFO:teuthology.orchestra.run.smithi006.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:15:26.609 INFO:teuthology.orchestra.run.smithi006.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:15:26.609 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:15:26.609 INFO:teuthology.orchestra.run.smithi006.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:15:26.610 INFO:teuthology.lock.ops:Updating smithi006.front.sepia.ceph.com on lock server 2023-09-30T00:15:26.631 DEBUG:teuthology.orchestra.run.smithi022:> uname -m 2023-09-30T00:15:26.656 INFO:teuthology.orchestra.run.smithi022.stdout:x86_64 2023-09-30T00:15:26.656 DEBUG:teuthology.orchestra.run.smithi022:> cat /etc/os-release 2023-09-30T00:15:26.717 INFO:teuthology.orchestra.run.smithi022.stdout:NAME="CentOS Stream" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION="8" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:ID="centos" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:ID_LIKE="rhel fedora" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:VERSION_ID="8" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:PLATFORM_ID="platform:el8" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:PRETTY_NAME="CentOS Stream 8" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:ANSI_COLOR="0;31" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:HOME_URL="https://centos.org/" 2023-09-30T00:15:26.718 INFO:teuthology.orchestra.run.smithi022.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-09-30T00:15:26.719 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-09-30T00:15:26.719 INFO:teuthology.orchestra.run.smithi022.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-09-30T00:15:26.719 INFO:teuthology.lock.ops:Updating smithi022.front.sepia.ceph.com on lock server 2023-09-30T00:15:26.739 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-09-30T00:15:26.747 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-09-30T00:15:26.752 INFO:teuthology.task.internal:Checking for old test directory... 2023-09-30T00:15:26.753 DEBUG:teuthology.orchestra.run.smithi006:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:15:26.755 DEBUG:teuthology.orchestra.run.smithi022:> test '!' -e /home/ubuntu/cephtest 2023-09-30T00:15:26.777 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-09-30T00:15:26.783 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-09-30T00:15:26.783 DEBUG:teuthology.orchestra.run.smithi006:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:15:26.817 DEBUG:teuthology.orchestra.run.smithi022:> test -z $(ls -A /var/lib/ceph) 2023-09-30T00:15:26.852 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-09-30T00:15:26.907 INFO:teuthology.run_tasks:Running task kernel... 2023-09-30T00:15:26.921 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-09-30T00:15:26.921 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-09-30T00:15:26.922 DEBUG:teuthology.orchestra.run.smithi006:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:15:26.922 DEBUG:teuthology.orchestra.run.smithi022:> test -f /run/.containerenv -o -f /.dockerenv 2023-09-30T00:15:26.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:15:26.942 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2023-09-30T00:15:26.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:15:26.943 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-09-30T00:15:27.002 INFO:teuthology.orchestra.run.smithi022.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:15:27.002 INFO:teuthology.task.kernel:Running kernel on smithi022: 4.18.0-489.el8.x86_64 2023-09-30T00:15:27.003 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum install -y kernel 2023-09-30T00:15:27.006 INFO:teuthology.orchestra.run.smithi006.stdout:4.18.0-489.el8.x86_64 2023-09-30T00:15:27.006 INFO:teuthology.task.kernel:Running kernel on smithi006: 4.18.0-489.el8.x86_64 2023-09-30T00:15:27.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum install -y kernel 2023-09-30T00:25:53.709 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - AppStream 55 kB/s | 33 MB 10:24 2023-09-30T00:26:06.076 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - BaseOS 8.1 MB/s | 48 MB 00:05 2023-09-30T00:26:13.650 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - Extras 88 kB/s | 18 kB 00:00 2023-09-30T00:26:14.293 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - Extras common packages 15 kB/s | 6.9 kB 00:00 2023-09-30T00:26:14.711 INFO:teuthology.orchestra.run.smithi006.stdout:Copr repo for python3-asyncssh owned by ceph 14 kB/s | 3.5 kB 00:00 2023-09-30T00:26:15.628 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 21 MB/s | 16 MB 00:00 2023-09-30T00:26:19.670 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 331 kB/s | 24 kB 00:00 2023-09-30T00:26:23.487 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:26:23.487 INFO:teuthology.orchestra.run.smithi006.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:26:23.743 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:26:23.744 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:26:23.744 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-09-30T00:26:23.744 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout:Install 3 Packages 2023-09-30T00:26:23.745 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:26:23.746 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 88 M 2023-09-30T00:26:23.746 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 96 M 2023-09-30T00:26:23.746 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:26:24.430 INFO:teuthology.orchestra.run.smithi006.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 16 MB/s | 10 MB 00:00 2023-09-30T00:26:25.297 INFO:teuthology.orchestra.run.smithi006.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 23 MB/s | 35 MB 00:01 2023-09-30T00:26:25.789 INFO:teuthology.orchestra.run.smithi006.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 21 MB/s | 43 MB 00:02 2023-09-30T00:26:25.789 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:26:25.790 INFO:teuthology.orchestra.run.smithi006.stdout:Total 43 MB/s | 88 MB 00:02 2023-09-30T00:26:26.495 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:26:26.957 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:26:26.957 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:26:28.249 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:26:28.271 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:26:31.228 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:26:31.779 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:26:35.498 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:26:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:26:39.564 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:26:39.888 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:27:03.200 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:27:03.223 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:27:03.800 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:27:03.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:27:03.800 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:27:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:27:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:27:04.557 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:27:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:27:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:27:04.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:27:04.558 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:27:04.697 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:27:04.918 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:27:04.919 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:27:04.919 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:27:04.919 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:27:04.919 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:27:04.919 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2023-09-30T00:27:04.919 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi006.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:27:04.920 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:27:05.138 INFO:teuthology.orchestra.run.smithi006.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:27:05.138 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:27:05.138 DEBUG:teuthology.orchestra.run.smithi006:> sudo rpm -qi grub2-tools 2023-09-30T00:27:05.173 INFO:teuthology.orchestra.run.smithi006.stdout:Name : grub2-tools 2023-09-30T00:27:05.173 INFO:teuthology.orchestra.run.smithi006.stdout:Epoch : 1 2023-09-30T00:27:05.173 INFO:teuthology.orchestra.run.smithi006.stdout:Version : 2.02 2023-09-30T00:27:05.173 INFO:teuthology.orchestra.run.smithi006.stdout:Release : 129.el8 2023-09-30T00:27:05.173 INFO:teuthology.orchestra.run.smithi006.stdout:Architecture: x86_64 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Group : System Environment/Base 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Size : 9265586 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:License : GPLv3+ 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Relocations : (not relocatable) 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Packager : CentOS BuildSystem 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:Vendor : CentOS 2023-09-30T00:27:05.174 INFO:teuthology.orchestra.run.smithi006.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:Summary : Support tools for GRUB. 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:Description : 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:hardware devices. 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:27:05.175 INFO:teuthology.orchestra.run.smithi006.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:27:05.176 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:27:05.176 INFO:teuthology.task.kernel:Updating grub on smithi006 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:27:05.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:27:06.759 INFO:teuthology.orchestra.run.smithi006.stderr:Generating grub configuration file ... 2023-09-30T00:27:08.955 INFO:teuthology.orchestra.run.smithi006.stderr:done 2023-09-30T00:27:08.958 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-09-30T00:27:08.987 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.BAu7yb5Jst 2023-09-30T00:27:08.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.BAu7yb5Jst 2023-09-30T00:27:09.060 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.BAu7yb5Jst 2023-09-30T00:27:09.214 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.BAu7yb5Jst is 6KB 2023-09-30T00:27:09.264 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.BAu7yb5Jst 2023-09-30T00:27:09.282 DEBUG:teuthology.orchestra.run.smithi006:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:27:09.367 INFO:teuthology.orchestra.run.smithi006.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:27:09.367 INFO:teuthology.orchestra.run.smithi006.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:27:09.367 INFO:teuthology.orchestra.run.smithi006.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:27:09.367 INFO:teuthology.orchestra.run.smithi006.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:27:09.368 DEBUG:teuthology.orchestra.run.smithi006:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:27:09.502 DEBUG:teuthology.orchestra.run.smithi006:> sudo shutdown -r now 2023-09-30T00:27:39.505 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2023-09-30T00:27:39.505 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-09-30T00:27:39.506 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:27:52.388 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - AppStream 46 kB/s | 33 MB 12:22 2023-09-30T00:27:57.946 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.6 2023-09-30T00:28:05.287 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - BaseOS 7.3 MB/s | 48 MB 00:06 2023-09-30T00:28:06.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-09-30T00:28:06.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:28:13.087 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2023-09-30T00:28:13.696 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-09-30T00:28:14.014 INFO:teuthology.orchestra.run.smithi022.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-09-30T00:28:15.917 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 9.1 MB/s | 16 MB 00:01 2023-09-30T00:28:19.966 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 410 kB/s | 24 kB 00:00 2023-09-30T00:28:23.707 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-09-30T00:28:23.707 INFO:teuthology.orchestra.run.smithi022.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-09-30T00:28:23.979 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-09-30T00:28:23.980 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-09-30T00:28:23.981 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:28:23.981 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:28:23.981 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:28:23.981 INFO:teuthology.orchestra.run.smithi022.stdout:Install 3 Packages 2023-09-30T00:28:23.981 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:28:23.981 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 88 M 2023-09-30T00:28:23.982 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 96 M 2023-09-30T00:28:23.982 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:28:24.600 INFO:teuthology.orchestra.run.smithi022.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 17 MB/s | 10 MB 00:00 2023-09-30T00:28:25.509 INFO:teuthology.orchestra.run.smithi022.stdout:(2/3): kernel-core-4.18.0-514.el8.x86_64.rpm 29 MB/s | 43 MB 00:01 2023-09-30T00:28:25.826 INFO:teuthology.orchestra.run.smithi022.stdout:(3/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 20 MB/s | 35 MB 00:01 2023-09-30T00:28:25.826 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:28:25.826 INFO:teuthology.orchestra.run.smithi022.stdout:Total 48 MB/s | 88 MB 00:01 2023-09-30T00:28:26.531 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:28:26.973 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:28:26.973 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:28:28.272 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:28:28.307 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:28:31.320 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:28:31.803 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:28:35.533 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:28:35.928 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:28:39.555 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:28:39.844 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:29:03.153 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:29:03.177 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:29:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:29:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-09-30T00:29:03.543 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-09-30T00:29:04.331 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-09-30T00:29:04.331 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:29:04.331 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:29:04.332 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-09-30T00:29:04.332 INFO:teuthology.orchestra.run.smithi022.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-09-30T00:29:04.332 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:29:04.332 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:29:04.479 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:29:04.713 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:29:04.713 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:29:04.713 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-09-30T00:29:04.713 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-09-30T00:29:04.714 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-09-30T00:29:04.714 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2023-09-30T00:29:04.714 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi022.front.sepia.ceph.com, path=None, version=distro) 2023-09-30T00:29:04.714 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q kernel | sort -rV | head -n 1 2023-09-30T00:29:04.932 INFO:teuthology.orchestra.run.smithi022.stdout:kernel-4.18.0-514.el8.x86_64 2023-09-30T00:29:04.932 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-09-30T00:29:04.933 DEBUG:teuthology.orchestra.run.smithi022:> sudo rpm -qi grub2-tools 2023-09-30T00:29:04.967 INFO:teuthology.orchestra.run.smithi022.stdout:Name : grub2-tools 2023-09-30T00:29:04.967 INFO:teuthology.orchestra.run.smithi022.stdout:Epoch : 1 2023-09-30T00:29:04.967 INFO:teuthology.orchestra.run.smithi022.stdout:Version : 2.02 2023-09-30T00:29:04.967 INFO:teuthology.orchestra.run.smithi022.stdout:Release : 129.el8 2023-09-30T00:29:04.967 INFO:teuthology.orchestra.run.smithi022.stdout:Architecture: x86_64 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Group : System Environment/Base 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Size : 9265586 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:License : GPLv3+ 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Build Host : kbuilder.bsys.centos.org 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Relocations : (not relocatable) 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Packager : CentOS BuildSystem 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:Vendor : CentOS 2023-09-30T00:29:04.968 INFO:teuthology.orchestra.run.smithi022.stdout:URL : http://www.gnu.org/software/grub/ 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:Summary : Support tools for GRUB. 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:Description : 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:customizable bootloader with modular architecture. It supports a rich 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:variety of kernel formats, file systems, computer architectures and 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:hardware devices. 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:29:04.969 INFO:teuthology.orchestra.run.smithi022.stdout:This subpackage provides tools for support of all platforms. 2023-09-30T00:29:04.970 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-09-30T00:29:04.970 INFO:teuthology.task.kernel:Updating grub on smithi022 to boot 4.18.0-514.el8.x86_64 2023-09-30T00:29:04.970 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-09-30T00:29:06.257 INFO:teuthology.orchestra.run.smithi022.stderr:Generating grub configuration file ... 2023-09-30T00:29:06.950 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:29:08.251 INFO:teuthology.orchestra.run.smithi022.stderr:done 2023-09-30T00:29:08.262 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2023-09-30T00:29:08.290 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.IsZic3AnA4 2023-09-30T00:29:08.291 DEBUG:teuthology.orchestra.run.smithi022:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.IsZic3AnA4 2023-09-30T00:29:08.362 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.IsZic3AnA4 2023-09-30T00:29:08.568 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.IsZic3AnA4 is 6KB 2023-09-30T00:29:08.617 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.IsZic3AnA4 2023-09-30T00:29:08.635 DEBUG:teuthology.orchestra.run.smithi022:> sudo /bin/ls /boot/loader/entries || true 2023-09-30T00:29:08.717 INFO:teuthology.orchestra.run.smithi022.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-09-30T00:29:08.717 INFO:teuthology.orchestra.run.smithi022.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-09-30T00:29:08.717 INFO:teuthology.orchestra.run.smithi022.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-09-30T00:29:08.717 INFO:teuthology.orchestra.run.smithi022.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-09-30T00:29:08.719 DEBUG:teuthology.orchestra.run.smithi022:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-09-30T00:29:08.882 DEBUG:teuthology.orchestra.run.smithi022:> sudo shutdown -r now 2023-09-30T00:29:18.958 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-09-30T00:29:18.959 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:29:19.305 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-09-30T00:29:19.633 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-09-30T00:29:19.633 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:29:19.634 DEBUG:teuthology.orchestra.run.smithi006:> uname -r 2023-09-30T00:29:19.656 INFO:teuthology.orchestra.run.smithi006.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:29:19.656 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:29:19.656 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:29:19.656 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:29:20.657 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2023-09-30T00:29:20.657 DEBUG:teuthology.orchestra.run.smithi006:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:29:20.737 INFO:teuthology.orchestra.run.smithi006.stdout:ttyS1 2023-09-30T00:29:20.750 DEBUG:teuthology.parallel:result is None 2023-09-30T00:29:38.885 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2023-09-30T00:29:38.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-09-30T00:29:38.886 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:29:57.242 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.22 2023-09-30T00:30:06.244 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-09-30T00:30:06.244 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:31:06.247 DEBUG:teuthology.orchestra.remote:timed out 2023-09-30T00:31:18.247 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-09-30T00:31:18.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:31:18.581 DEBUG:teuthology.orchestra.run.smithi022:> true 2023-09-30T00:31:18.878 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-09-30T00:31:18.878 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-514.el8.x86_64"... 2023-09-30T00:31:18.878 DEBUG:teuthology.orchestra.run.smithi022:> uname -r 2023-09-30T00:31:18.900 INFO:teuthology.orchestra.run.smithi022.stdout:4.18.0-514.el8.x86_64 2023-09-30T00:31:18.901 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-09-30T00:31:18.901 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-09-30T00:31:18.901 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-09-30T00:31:19.902 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2023-09-30T00:31:19.902 DEBUG:teuthology.orchestra.run.smithi022:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-09-30T00:31:19.980 INFO:teuthology.orchestra.run.smithi022.stdout:ttyS1 2023-09-30T00:31:19.992 DEBUG:teuthology.parallel:result is None 2023-09-30T00:31:19.993 INFO:teuthology.run_tasks:Running task internal.base... 2023-09-30T00:31:20.006 INFO:teuthology.task.internal:Creating test directory... 2023-09-30T00:31:20.006 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:31:20.010 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-09-30T00:31:20.033 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-09-30T00:31:20.040 INFO:teuthology.run_tasks:Running task internal.archive... 2023-09-30T00:31:20.047 INFO:teuthology.task.internal:Creating archive directory... 2023-09-30T00:31:20.047 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:31:20.074 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-09-30T00:31:20.120 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-09-30T00:31:20.127 INFO:teuthology.task.internal:Enabling coredump saving... 2023-09-30T00:31:20.127 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:31:20.141 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-09-30T00:31:20.184 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:31:20.195 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:31:20.196 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:31:20.208 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-09-30T00:31:20.210 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-09-30T00:31:20.217 INFO:teuthology.task.internal:Configuring sudo... 2023-09-30T00:31:20.217 DEBUG:teuthology.orchestra.run.smithi006:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:31:20.220 DEBUG:teuthology.orchestra.run.smithi022:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-09-30T00:31:20.285 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-09-30T00:31:20.295 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-09-30T00:31:20.296 DEBUG:teuthology.orchestra.run.smithi006:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:31:20.300 DEBUG:teuthology.orchestra.run.smithi022:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-09-30T00:31:20.348 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:31:20.413 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:31:20.464 DEBUG:teuthology.orchestra.run.smithi006:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:31:20.555 DEBUG:teuthology.orchestra.run.smithi006:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:31:20.606 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:31:20.606 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:31:20.682 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:31:20.732 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-09-30T00:31:20.790 DEBUG:teuthology.orchestra.run.smithi022:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:31:20.879 DEBUG:teuthology.orchestra.run.smithi022:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-09-30T00:31:20.931 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:31:20.931 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-09-30T00:31:21.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo service rsyslog restart 2023-09-30T00:31:21.009 DEBUG:teuthology.orchestra.run.smithi022:> sudo service rsyslog restart 2023-09-30T00:31:21.097 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:31:21.108 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:31:21.481 INFO:teuthology.run_tasks:Running task internal.timer... 2023-09-30T00:31:21.491 INFO:teuthology.task.internal:Starting timer... 2023-09-30T00:31:21.492 INFO:teuthology.run_tasks:Running task pcp... 2023-09-30T00:31:21.501 INFO:teuthology.run_tasks:Running task selinux... 2023-09-30T00:31:21.509 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-09-30T00:31:21.509 DEBUG:teuthology.orchestra.run.smithi006:> sudo service auditd rotate 2023-09-30T00:31:21.591 INFO:teuthology.orchestra.run.smithi006.stdout:Rotating logs: 2023-09-30T00:31:21.593 DEBUG:teuthology.orchestra.run.smithi022:> sudo service auditd rotate 2023-09-30T00:31:21.684 INFO:teuthology.orchestra.run.smithi022.stdout:Rotating logs: 2023-09-30T00:31:21.686 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-09-30T00:31:21.686 DEBUG:teuthology.orchestra.run.smithi006:> /usr/sbin/getenforce 2023-09-30T00:31:21.714 INFO:teuthology.orchestra.run.smithi006.stdout:Permissive 2023-09-30T00:31:21.715 DEBUG:teuthology.orchestra.run.smithi022:> /usr/sbin/getenforce 2023-09-30T00:31:21.764 INFO:teuthology.orchestra.run.smithi022.stdout:Permissive 2023-09-30T00:31:21.764 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi006.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi022.front.sepia.ceph.com': 'permissive'} 2023-09-30T00:31:21.765 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:31:21.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:31:21.801 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:31:21.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:31:21.836 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-09-30T00:31:21.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:31:21.876 DEBUG:teuthology.orchestra.run.smithi022:> sudo /usr/sbin/setenforce permissive 2023-09-30T00:31:21.915 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-09-30T00:31:21.924 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-09-30T00:31:21.925 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-09-30T00:31:21.942 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-09-30T00:31:21.943 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi006.front.sepia.ceph.com,smithi022.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-09-30T00:39:32.068 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi006.front.sepia.ceph.com'), Remote(name='ubuntu@smithi022.front.sepia.ceph.com')] 2023-09-30T00:39:32.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-09-30T00:39:32.070 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi006.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:39:32.142 DEBUG:teuthology.orchestra.run.smithi006:> true 2023-09-30T00:39:32.222 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi006.front.sepia.ceph.com' 2023-09-30T00:39:32.222 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-09-30T00:39:32.222 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi022.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-09-30T00:39:32.292 DEBUG:teuthology.orchestra.run.smithi022:> true 2023-09-30T00:39:32.370 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi022.front.sepia.ceph.com' 2023-09-30T00:39:32.371 INFO:teuthology.run_tasks:Running task clock... 2023-09-30T00:39:32.383 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-09-30T00:39:32.383 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:39:32.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:39:32.386 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-09-30T00:39:32.386 DEBUG:teuthology.orchestra.run.smithi022:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:39:32.425 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:39:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:39:32.451 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-09-30T00:39:32.468 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-09-30T00:39:32.515 INFO:teuthology.orchestra.run.smithi006.stderr:sudo: ntpd: command not found 2023-09-30T00:39:32.529 INFO:teuthology.orchestra.run.smithi006.stdout:506 Cannot talk to daemon 2023-09-30T00:39:32.545 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:39:32.554 INFO:teuthology.orchestra.run.smithi022.stderr:sudo: ntpd: command not found 2023-09-30T00:39:32.561 INFO:teuthology.orchestra.run.smithi006.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:39:32.568 INFO:teuthology.orchestra.run.smithi022.stdout:506 Cannot talk to daemon 2023-09-30T00:39:32.586 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-09-30T00:39:32.602 INFO:teuthology.orchestra.run.smithi022.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-09-30T00:39:32.636 INFO:teuthology.orchestra.run.smithi006.stderr:bash: ntpq: command not found 2023-09-30T00:39:32.640 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:39:32.640 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2023-09-30T00:39:32.640 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.640 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.641 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.641 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.679 INFO:teuthology.orchestra.run.smithi022.stderr:bash: ntpq: command not found 2023-09-30T00:39:32.683 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:39:32.683 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2023-09-30T00:39:32.683 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.683 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.684 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.684 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:39:32.685 INFO:teuthology.run_tasks:Running task install... 2023-09-30T00:39:32.697 DEBUG:teuthology.task.install:project ceph 2023-09-30T00:39:32.697 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}} 2023-09-30T00:39:32.697 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'} 2023-09-30T00:39:32.697 INFO:teuthology.task.install:Using flavor: default 2023-09-30T00:39:32.703 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-09-30T00:39:32.703 INFO:teuthology.task.install:extra packages: [] 2023-09-30T00:39:32.704 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:39:32.704 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:39:32.706 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'tag': None, 'wait_for_package': False} 2023-09-30T00:39:32.706 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:39:32.851 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:39:32.851 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:39:32.857 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/ 2023-09-30T00:39:32.858 INFO:teuthology.task.install.rpm:Package version is 17.2.6-1165.g93dce5e1 2023-09-30T00:39:33.157 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:39:33.158 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:39:33.158 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:39:33.194 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:39:33.195 DEBUG:teuthology.orchestra.run.smithi006:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:39:33.224 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-yuri4-testing-2023-09-29-0928-quincy/93dce5e17918f8aca65993a7a2dc40233a47b481/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-09-30T00:39:33.224 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:39:33.224 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-09-30T00:39:33.261 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-09-30T00:39:33.261 DEBUG:teuthology.orchestra.run.smithi022:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/93dce5e17918f8aca65993a7a2dc40233a47b481/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-09-30T00:39:33.271 DEBUG:teuthology.orchestra.run.smithi006:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:39:33.338 DEBUG:teuthology.orchestra.run.smithi022:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-09-30T00:39:33.359 DEBUG:teuthology.orchestra.run.smithi006:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:39:33.393 INFO:teuthology.orchestra.run.smithi006.stdout:check_obsoletes = 1 2023-09-30T00:39:33.394 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum clean all 2023-09-30T00:39:33.427 DEBUG:teuthology.orchestra.run.smithi022:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-09-30T00:39:33.460 INFO:teuthology.orchestra.run.smithi022.stdout:check_obsoletes = 1 2023-09-30T00:39:33.462 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum clean all 2023-09-30T00:39:33.701 INFO:teuthology.orchestra.run.smithi006.stdout:76 files removed 2023-09-30T00:39:33.733 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-radosgw 2023-09-30T00:39:33.771 INFO:teuthology.orchestra.run.smithi022.stdout:76 files removed 2023-09-30T00:39:33.804 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-radosgw 2023-09-30T00:39:35.142 INFO:teuthology.orchestra.run.smithi006.stdout:ceph packages for x86_64 84 kB/s | 78 kB 00:00 2023-09-30T00:39:35.148 INFO:teuthology.orchestra.run.smithi022.stdout:ceph packages for x86_64 89 kB/s | 78 kB 00:00 2023-09-30T00:39:35.791 INFO:teuthology.orchestra.run.smithi022.stdout:ceph noarch packages 15 kB/s | 9.2 kB 00:00 2023-09-30T00:39:35.794 INFO:teuthology.orchestra.run.smithi006.stdout:ceph noarch packages 15 kB/s | 9.2 kB 00:00 2023-09-30T00:39:36.429 INFO:teuthology.orchestra.run.smithi006.stdout:ceph source packages 2.4 kB/s | 1.5 kB 00:00 2023-09-30T00:39:36.466 INFO:teuthology.orchestra.run.smithi022.stdout:ceph source packages 2.3 kB/s | 1.5 kB 00:00 2023-09-30T00:39:38.824 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - AppStream 14 MB/s | 33 MB 00:02 2023-09-30T00:39:38.856 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - AppStream 14 MB/s | 33 MB 00:02 2023-09-30T00:39:47.004 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - BaseOS 33 MB/s | 48 MB 00:01 2023-09-30T00:39:47.266 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 48 MB 00:01 2023-09-30T00:39:54.946 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - Extras 72 kB/s | 18 kB 00:00 2023-09-30T00:39:55.358 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - Extras 45 kB/s | 18 kB 00:00 2023-09-30T00:39:55.705 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - Extras common packages 12 kB/s | 6.9 kB 00:00 2023-09-30T00:39:55.994 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - Extras common packages 16 kB/s | 6.9 kB 00:00 2023-09-30T00:39:57.931 INFO:teuthology.orchestra.run.smithi006.stdout:CentOS Stream 8 - PowerTools 3.2 MB/s | 6.5 MB 00:02 2023-09-30T00:39:58.093 INFO:teuthology.orchestra.run.smithi022.stdout:CentOS Stream 8 - PowerTools 3.4 MB/s | 6.5 MB 00:01 2023-09-30T00:39:59.450 INFO:teuthology.orchestra.run.smithi006.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-09-30T00:39:59.645 INFO:teuthology.orchestra.run.smithi022.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-09-30T00:40:00.317 INFO:teuthology.orchestra.run.smithi006.stdout:Extra Packages for Enterprise Linux 23 MB/s | 16 MB 00:00 2023-09-30T00:40:00.746 INFO:teuthology.orchestra.run.smithi022.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-09-30T00:40:04.684 INFO:teuthology.orchestra.run.smithi006.stdout:lab-extras 439 kB/s | 24 kB 00:00 2023-09-30T00:40:05.171 INFO:teuthology.orchestra.run.smithi022.stdout:lab-extras 277 kB/s | 24 kB 00:00 2023-09-30T00:40:09.622 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:40:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:40:09.625 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:40:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:40:09.625 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrading: 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:40:09.626 INFO:teuthology.orchestra.run.smithi006.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T00:40:09.627 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:Install 21 Packages 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout:Upgrade 2 Packages 2023-09-30T00:40:09.628 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:40:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 63 M 2023-09-30T00:40:09.629 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:40:10.059 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:40:10.062 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:40:10.062 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrading: 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout: librados2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.6 M 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.2 M 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 23 M 2023-09-30T00:40:10.063 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 24 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 796 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 506 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2 x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 5.6 M 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-09-30T00:40:10.064 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 45 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 115 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 190 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 370 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 358 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 107 k 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: thrift x86_64 0.13.0-2.el8 epel 1.7 M 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:40:10.065 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:40:10.066 INFO:teuthology.orchestra.run.smithi022.stdout:Install 21 Packages 2023-09-30T00:40:10.066 INFO:teuthology.orchestra.run.smithi022.stdout:Upgrade 2 Packages 2023-09-30T00:40:10.066 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:40:10.066 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 63 M 2023-09-30T00:40:10.067 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:40:14.710 INFO:teuthology.orchestra.run.smithi006.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.2 MB/s | 5.2 MB 00:04 2023-09-30T00:40:14.810 INFO:teuthology.orchestra.run.smithi006.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 245 kB/s | 24 kB 00:00 2023-09-30T00:40:15.569 INFO:teuthology.orchestra.run.smithi006.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 1.0 MB/s | 796 kB 00:00 2023-09-30T00:40:15.928 INFO:teuthology.orchestra.run.smithi006.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.4 MB/s | 506 kB 00:00 2023-09-30T00:40:16.022 INFO:teuthology.orchestra.run.smithi022.stdout:(1/23): ceph-base-17.2.6-1165.g93dce5e1.el8.x86 1.4 MB/s | 5.2 MB 00:03 2023-09-30T00:40:16.105 INFO:teuthology.orchestra.run.smithi022.stdout:(2/23): ceph-selinux-17.2.6-1165.g93dce5e1.el8. 293 kB/s | 24 kB 00:00 2023-09-30T00:40:16.270 INFO:teuthology.orchestra.run.smithi006.stdout:(5/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 4.0 MB/s | 23 MB 00:05 2023-09-30T00:40:16.378 INFO:teuthology.orchestra.run.smithi006.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 412 kB/s | 45 kB 00:00 2023-09-30T00:40:16.487 INFO:teuthology.orchestra.run.smithi006.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.0 MB/s | 115 kB 00:00 2023-09-30T00:40:16.604 INFO:teuthology.orchestra.run.smithi006.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 1.6 MB/s | 190 kB 00:00 2023-09-30T00:40:16.639 INFO:teuthology.orchestra.run.smithi022.stdout:(3/23): libcephfs2-17.2.6-1165.g93dce5e1.el8.x8 1.5 MB/s | 796 kB 00:00 2023-09-30T00:40:16.804 INFO:teuthology.orchestra.run.smithi006.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 1.8 MB/s | 370 kB 00:00 2023-09-30T00:40:17.047 INFO:teuthology.orchestra.run.smithi022.stdout:(4/23): libradosstriper1-17.2.6-1165.g93dce5e1. 1.2 MB/s | 506 kB 00:00 2023-09-30T00:40:17.587 INFO:teuthology.orchestra.run.smithi006.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 457 kB/s | 358 kB 00:00 2023-09-30T00:40:17.696 INFO:teuthology.orchestra.run.smithi006.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 986 kB/s | 107 kB 00:00 2023-09-30T00:40:17.921 INFO:teuthology.orchestra.run.smithi006.stdout:(12/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-09-30T00:40:18.014 INFO:teuthology.orchestra.run.smithi006.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.7 MB/s | 259 kB 00:00 2023-09-30T00:40:18.081 INFO:teuthology.orchestra.run.smithi006.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 665 kB/s | 44 kB 00:00 2023-09-30T00:40:18.248 INFO:teuthology.orchestra.run.smithi006.stdout:(15/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 282 kB/s | 47 kB 00:00 2023-09-30T00:40:18.873 INFO:teuthology.orchestra.run.smithi006.stdout:(16/23): gperftools-libs-2.7-9.el8.x86_64.rpm 490 kB/s | 306 kB 00:00 2023-09-30T00:40:19.023 INFO:teuthology.orchestra.run.smithi006.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 393 kB/s | 59 kB 00:00 2023-09-30T00:40:19.140 INFO:teuthology.orchestra.run.smithi006.stdout:(18/23): libunwind-1.3.1-3.el8.x86_64.rpm 643 kB/s | 75 kB 00:00 2023-09-30T00:40:19.474 INFO:teuthology.orchestra.run.smithi006.stdout:(19/23): thrift-0.13.0-2.el8.x86_64.rpm 5.2 MB/s | 1.7 MB 00:00 2023-09-30T00:40:19.497 INFO:teuthology.orchestra.run.smithi022.stdout:(5/23): ceph-common-17.2.6-1165.g93dce5e1.el8.x 3.2 MB/s | 23 MB 00:07 2023-09-30T00:40:19.589 INFO:teuthology.orchestra.run.smithi022.stdout:(6/23): python3-ceph-argparse-17.2.6-1165.g93dc 489 kB/s | 45 kB 00:00 2023-09-30T00:40:19.641 INFO:teuthology.orchestra.run.smithi006.stdout:(20/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 1.8 MB/s | 17 MB 00:09 2023-09-30T00:40:19.673 INFO:teuthology.orchestra.run.smithi022.stdout:(7/23): python3-ceph-common-17.2.6-1165.g93dce5 1.4 MB/s | 115 kB 00:00 2023-09-30T00:40:19.766 INFO:teuthology.orchestra.run.smithi022.stdout:(8/23): python3-cephfs-17.2.6-1165.g93dce5e1.el 2.0 MB/s | 190 kB 00:00 2023-09-30T00:40:19.915 INFO:teuthology.orchestra.run.smithi022.stdout:(9/23): python3-rados-17.2.6-1165.g93dce5e1.el8 2.4 MB/s | 370 kB 00:00 2023-09-30T00:40:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:(21/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 1.4 MB/s | 5.6 MB 00:03 2023-09-30T00:40:20.074 INFO:teuthology.orchestra.run.smithi022.stdout:(10/23): python3-rbd-17.2.6-1165.g93dce5e1.el8. 2.2 MB/s | 358 kB 00:00 2023-09-30T00:40:20.157 INFO:teuthology.orchestra.run.smithi022.stdout:(11/23): python3-rgw-17.2.6-1165.g93dce5e1.el8. 1.3 MB/s | 107 kB 00:00 2023-09-30T00:40:20.291 INFO:teuthology.orchestra.run.smithi022.stdout:(12/23): librgw2-17.2.6-1165.g93dce5e1.el8.x86_ 1.7 MB/s | 5.6 MB 00:03 2023-09-30T00:40:20.383 INFO:teuthology.orchestra.run.smithi022.stdout:(13/23): librdkafka-0.11.4-3.el8.x86_64.rpm 1.5 MB/s | 354 kB 00:00 2023-09-30T00:40:20.433 INFO:teuthology.orchestra.run.smithi022.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 883 kB/s | 44 kB 00:00 2023-09-30T00:40:20.535 INFO:teuthology.orchestra.run.smithi022.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.0 MB/s | 259 kB 00:00 2023-09-30T00:40:20.583 INFO:teuthology.orchestra.run.smithi022.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 312 kB/s | 47 kB 00:00 2023-09-30T00:40:20.641 INFO:teuthology.orchestra.run.smithi006.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.0 MB/s | 3.5 MB 00:01 2023-09-30T00:40:20.900 INFO:teuthology.orchestra.run.smithi022.stdout:(17/23): liboath-2.6.2-3.el8.x86_64.rpm 186 kB/s | 59 kB 00:00 2023-09-30T00:40:21.023 INFO:teuthology.orchestra.run.smithi022.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 628 kB/s | 306 kB 00:00 2023-09-30T00:40:21.076 INFO:teuthology.orchestra.run.smithi022.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 428 kB/s | 75 kB 00:00 2023-09-30T00:40:21.287 INFO:teuthology.orchestra.run.smithi006.stdout:(23/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 2.2 MB/s | 3.6 MB 00:01 2023-09-30T00:40:21.291 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:40:21.291 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.4 MB/s | 63 MB 00:11 2023-09-30T00:40:21.326 INFO:teuthology.orchestra.run.smithi022.stdout:(20/23): thrift-0.13.0-2.el8.x86_64.rpm 5.7 MB/s | 1.7 MB 00:00 2023-09-30T00:40:21.337 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:40:21.390 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:40:21.390 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:40:21.801 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:40:21.804 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:40:22.018 INFO:teuthology.orchestra.run.smithi022.stdout:(21/23): librbd1-17.2.6-1165.g93dce5e1.el8.x86_ 5.2 MB/s | 3.6 MB 00:00 2023-09-30T00:40:22.152 INFO:teuthology.orchestra.run.smithi022.stdout:(22/23): librados2-17.2.6-1165.g93dce5e1.el8.x8 3.2 MB/s | 3.5 MB 00:01 2023-09-30T00:40:22.769 INFO:teuthology.orchestra.run.smithi022.stdout:(23/23): ceph-radosgw-17.2.6-1165.g93dce5e1.el8 1.6 MB/s | 17 MB 00:10 2023-09-30T00:40:22.771 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:40:22.772 INFO:teuthology.orchestra.run.smithi022.stdout:Total 5.0 MB/s | 63 MB 00:12 2023-09-30T00:40:22.816 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:40:22.868 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:40:22.868 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:40:22.936 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:40:23.183 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:40:23.281 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:40:23.285 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:40:23.371 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:40:23.778 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:40:23.988 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:40:24.087 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:40:24.387 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:40:24.465 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:40:24.562 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:40:24.662 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-09-30T00:40:24.801 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:40:25.010 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:40:25.187 INFO:teuthology.orchestra.run.smithi006.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:40:25.204 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-09-30T00:40:25.421 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:40:25.426 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:40:25.514 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:40:25.643 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:40:25.804 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:40:25.839 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:40:25.984 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:40:26.004 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:40:26.435 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:40:26.602 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:40:26.609 INFO:teuthology.orchestra.run.smithi022.stdout: Upgrading : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:40:26.847 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/25 2023-09-30T00:40:26.937 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:40:27.012 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:40:27.039 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : thrift-0.13.0-2.el8.x86_64 7/25 2023-09-30T00:40:27.200 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:40:27.236 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 8/25 2023-09-30T00:40:27.392 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:40:27.434 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:40:27.618 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:40:27.846 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:40:28.012 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:40:28.044 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 9/25 2023-09-30T00:40:28.086 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:40:28.209 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:40:28.265 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:40:28.276 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:40:28.447 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:40:28.463 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 11/25 2023-09-30T00:40:28.621 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:40:28.647 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:40:28.779 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:40:28.882 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:40:29.085 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 14/25 2023-09-30T00:40:29.226 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:40:29.322 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 15/25 2023-09-30T00:40:29.515 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-09-30T00:40:29.669 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-09-30T00:40:29.892 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-09-30T00:40:30.051 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 19/25 2023-09-30T00:40:31.798 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:40:32.029 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:40:32.676 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:40:32.920 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:40:32.993 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:40:33.075 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:40:33.226 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:40:33.228 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:40:33.879 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 20/25 2023-09-30T00:40:34.144 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:40:34.270 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 21/25 2023-09-30T00:40:34.408 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:40:54.561 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:40:54.561 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /sys 2023-09-30T00:40:54.561 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /proc 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /dev 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /run 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /mnt 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /var/tmp 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /home 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /tmp 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout:skipping the directory /dev 2023-09-30T00:40:54.562 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:40:55.504 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /sys 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /proc 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /dev 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /run 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /mnt 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /var/tmp 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /home 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /tmp 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout:skipping the directory /dev 2023-09-30T00:40:55.505 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:40:56.434 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:40:56.501 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:40:56.502 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:40:56.728 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:40:56.775 INFO:teuthology.orchestra.run.smithi006.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:40:57.374 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:40:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 23/25 2023-09-30T00:40:57.444 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:40:57.504 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:40:57.505 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T00:40:57.506 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T00:40:57.507 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:40:57.507 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:40:57.507 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:40:57.649 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-09-30T00:40:57.649 INFO:teuthology.orchestra.run.smithi022.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:40:58.434 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_6 6/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 7/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8. 8/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x8 9/25 2023-09-30T00:40:58.435 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 10/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 11/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 12/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 13/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 14/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 19/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 20/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : thrift-0.13.0-2.el8.x86_64 21/25 2023-09-30T00:40:58.436 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 22/25 2023-09-30T00:40:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-09-30T00:40:58.437 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 24/25 2023-09-30T00:40:58.788 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:40:58.788 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:40:58.788 INFO:teuthology.orchestra.run.smithi006.stdout:Upgraded: 2023-09-30T00:40:58.788 INFO:teuthology.orchestra.run.smithi006.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.789 INFO:teuthology.orchestra.run.smithi006.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.790 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:58.791 INFO:teuthology.orchestra.run.smithi006.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:40:58.791 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:40:58.791 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:40:59.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-test 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout:Upgraded: 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout: librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout: librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:40:59.552 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-radosgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-selinux-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: liboath-2.6.2-3.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: libradosstriper1-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-09-30T00:40:59.553 INFO:teuthology.orchestra.run.smithi022.stdout: librgw2-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: libunwind-1.3.1-3.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-argparse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-ceph-common-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.554 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:40:59.555 INFO:teuthology.orchestra.run.smithi022.stdout: thrift-0.13.0-2.el8.x86_64 2023-09-30T00:40:59.555 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:40:59.555 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:40:59.812 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-test 2023-09-30T00:41:00.062 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:00:56 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:41:00.799 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:00:55 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:41:01.818 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:41:01.819 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:01.819 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:41:01.819 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:01.819 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:41:01.820 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:01.821 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2023-09-30T00:41:01.821 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:01.821 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 53 M 2023-09-30T00:41:01.821 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 233 M 2023-09-30T00:41:01.821 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:41:02.221 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 945 kB/s | 202 kB 00:00 2023-09-30T00:41:02.271 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2023-09-30T00:41:02.329 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 5.4 MB/s | 323 kB 00:00 2023-09-30T00:41:02.388 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T00:41:02.538 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 328 kB/s | 175 kB 00:00 2023-09-30T00:41:02.628 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:41:02.629 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:02.629 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:41:02.629 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:02.629 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 52 M 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: jq x86_64 1.6-7.el8 appstream 202 k 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 175 k 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-09-30T00:41:02.630 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:02.631 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:41:02.631 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:02.631 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2023-09-30T00:41:02.631 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:02.631 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 53 M 2023-09-30T00:41:02.632 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 233 M 2023-09-30T00:41:02.632 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:41:02.872 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): jq-1.6-7.el8.x86_64.rpm 1.0 MB/s | 202 kB 00:00 2023-09-30T00:41:02.930 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 3.2 MB/s | 187 kB 00:00 2023-09-30T00:41:03.006 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 4.2 MB/s | 323 kB 00:00 2023-09-30T00:41:03.065 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2023-09-30T00:41:03.115 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): libcephsqlite-17.2.6-1165.g93dce5e1.el8. 395 kB/s | 175 kB 00:00 2023-09-30T00:41:09.855 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 6.6 MB/s | 52 MB 00:07 2023-09-30T00:41:09.856 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:09.857 INFO:teuthology.orchestra.run.smithi006.stdout:Total 6.5 MB/s | 53 MB 00:08 2023-09-30T00:41:09.903 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:41:09.927 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:41:09.927 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:41:10.269 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:41:10.273 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:41:10.281 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-test-17.2.6-1165.g93dce5e1.el8.x86_ 6.8 MB/s | 52 MB 00:07 2023-09-30T00:41:10.283 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:10.283 INFO:teuthology.orchestra.run.smithi022.stdout:Total 6.9 MB/s | 53 MB 00:07 2023-09-30T00:41:10.330 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:41:10.353 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:41:10.353 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:41:10.701 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:41:10.705 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:41:10.786 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:41:11.047 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T00:41:11.246 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T00:41:11.263 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:41:11.365 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:41:11.445 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:41:11.532 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-09-30T00:41:11.628 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T00:41:11.733 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-09-30T00:41:11.790 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:41:11.885 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:41:11.972 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-09-30T00:41:12.154 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : jq-1.6-7.el8.x86_64 4/6 2023-09-30T00:41:12.326 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:41:17.311 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:41:17.572 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:41:17.816 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/6 2023-09-30T00:41:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:41:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:41:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:41:17.902 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T00:41:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T00:41:17.903 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T00:41:18.018 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:41:18.387 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 6/6 2023-09-30T00:41:18.387 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/6 2023-09-30T00:41:18.387 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/6 2023-09-30T00:41:18.387 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : jq-1.6-7.el8.x86_64 3/6 2023-09-30T00:41:18.388 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-09-30T00:41:18.388 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-09-30T00:41:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T00:41:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:18.701 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:41:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:18.701 INFO:teuthology.orchestra.run.smithi006.stdout: jq-1.6-7.el8.x86_64 2023-09-30T00:41:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:41:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T00:41:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:41:18.702 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:18.702 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:41:19.006 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph 2023-09-30T00:41:19.147 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-09-30T00:41:19.147 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:19.148 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:41:19.148 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-test-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:19.148 INFO:teuthology.orchestra.run.smithi022.stdout: jq-1.6-7.el8.x86_64 2023-09-30T00:41:19.148 INFO:teuthology.orchestra.run.smithi022.stdout: libcephsqlite-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:19.148 INFO:teuthology.orchestra.run.smithi022.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-09-30T00:41:19.149 INFO:teuthology.orchestra.run.smithi022.stdout: socat-1.7.4.1-1.el8.x86_64 2023-09-30T00:41:19.149 INFO:teuthology.orchestra.run.smithi022.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-09-30T00:41:19.149 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:19.149 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:41:19.392 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph 2023-09-30T00:41:19.989 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:15 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:41:20.375 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:15 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:41:21.752 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:41:21.756 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:21.756 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:41:21.756 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:21.756 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:41:21.756 INFO:teuthology.orchestra.run.smithi006.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:41:21.757 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T00:41:21.758 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:41:21.759 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:21.760 INFO:teuthology.orchestra.run.smithi006.stdout:Install 32 Packages 2023-09-30T00:41:21.760 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:21.760 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 28 M 2023-09-30T00:41:21.760 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 110 M 2023-09-30T00:41:21.760 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:41:22.147 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:41:22.151 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:22.151 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:41:22.151 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:22.151 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:41:22.151 INFO:teuthology.orchestra.run.smithi022.stdout: ceph x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 6.4 k 2023-09-30T00:41:22.152 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:41:22.152 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 2.3 M 2023-09-30T00:41:22.152 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 1.6 M 2023-09-30T00:41:22.152 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 238 k 2023-09-30T00:41:22.152 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.9 M 2023-09-30T00:41:22.152 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 17 M 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-09-30T00:41:22.153 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-09-30T00:41:22.154 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-09-30T00:41:22.155 INFO:teuthology.orchestra.run.smithi022.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:22.156 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:41:22.157 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:22.157 INFO:teuthology.orchestra.run.smithi022.stdout:Install 32 Packages 2023-09-30T00:41:22.157 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:22.158 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 28 M 2023-09-30T00:41:22.158 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 110 M 2023-09-30T00:41:22.158 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:41:22.514 INFO:teuthology.orchestra.run.smithi006.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 29 kB/s | 6.4 kB 00:00 2023-09-30T00:41:22.540 INFO:teuthology.orchestra.run.smithi022.stdout:(1/32): ceph-17.2.6-1165.g93dce5e1.el8.x86_64.r 27 kB/s | 6.4 kB 00:00 2023-09-30T00:41:23.530 INFO:teuthology.orchestra.run.smithi006.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.3 MB/s | 1.6 MB 00:01 2023-09-30T00:41:23.707 INFO:teuthology.orchestra.run.smithi022.stdout:(2/32): ceph-mgr-17.2.6-1165.g93dce5e1.el8.x86_ 1.1 MB/s | 1.6 MB 00:01 2023-09-30T00:41:24.457 INFO:teuthology.orchestra.run.smithi022.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 1.1 MB/s | 2.3 MB 00:02 2023-09-30T00:41:24.632 INFO:teuthology.orchestra.run.smithi022.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 1.3 MB/s | 238 kB 00:00 2023-09-30T00:41:24.816 INFO:teuthology.orchestra.run.smithi022.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 856 kB/s | 157 kB 00:00 2023-09-30T00:41:24.891 INFO:teuthology.orchestra.run.smithi022.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 518 kB/s | 39 kB 00:00 2023-09-30T00:41:24.942 INFO:teuthology.orchestra.run.smithi022.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 2.0 MB/s | 103 kB 00:00 2023-09-30T00:41:25.000 INFO:teuthology.orchestra.run.smithi022.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 980 kB/s | 57 kB 00:00 2023-09-30T00:41:25.092 INFO:teuthology.orchestra.run.smithi022.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 4.9 MB/s | 457 kB 00:00 2023-09-30T00:41:25.206 INFO:teuthology.orchestra.run.smithi006.stdout:(3/32): ceph-mds-17.2.6-1165.g93dce5e1.el8.x86_ 808 kB/s | 2.3 MB 00:02 2023-09-30T00:41:25.357 INFO:teuthology.orchestra.run.smithi022.stdout:(10/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 944 kB/s | 249 kB 00:00 2023-09-30T00:41:25.485 INFO:teuthology.orchestra.run.smithi022.stdout:(11/32): python3-cryptography-3.2.1-6.el8.x86_6 4.9 MB/s | 641 kB 00:00 2023-09-30T00:41:25.539 INFO:teuthology.orchestra.run.smithi006.stdout:(4/32): ceph-mgr-modules-core-17.2.6-1165.g93dc 715 kB/s | 238 kB 00:00 2023-09-30T00:41:25.543 INFO:teuthology.orchestra.run.smithi022.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.8 MB/s | 109 kB 00:00 2023-09-30T00:41:25.606 INFO:teuthology.orchestra.run.smithi006.stdout:(5/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86_ 1.3 MB/s | 3.9 MB 00:03 2023-09-30T00:41:25.731 INFO:teuthology.orchestra.run.smithi006.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 311 kB/s | 39 kB 00:00 2023-09-30T00:41:25.815 INFO:teuthology.orchestra.run.smithi006.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 570 kB/s | 157 kB 00:00 2023-09-30T00:41:25.840 INFO:teuthology.orchestra.run.smithi006.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 952 kB/s | 103 kB 00:00 2023-09-30T00:41:25.874 INFO:teuthology.orchestra.run.smithi006.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 973 kB/s | 57 kB 00:00 2023-09-30T00:41:25.991 INFO:teuthology.orchestra.run.smithi006.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 3.0 MB/s | 457 kB 00:00 2023-09-30T00:41:26.125 INFO:teuthology.orchestra.run.smithi006.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 993 kB/s | 249 kB 00:00 2023-09-30T00:41:26.192 INFO:teuthology.orchestra.run.smithi006.stdout:(12/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2023-09-30T00:41:26.317 INFO:teuthology.orchestra.run.smithi006.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.9 MB/s | 641 kB 00:00 2023-09-30T00:41:26.334 INFO:teuthology.orchestra.run.smithi006.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 307 kB/s | 44 kB 00:00 2023-09-30T00:41:26.409 INFO:teuthology.orchestra.run.smithi006.stdout:(15/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 2.3 MB/s | 173 kB 00:00 2023-09-30T00:41:26.468 INFO:teuthology.orchestra.run.smithi006.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 6.5 MB/s | 384 kB 00:00 2023-09-30T00:41:26.493 INFO:teuthology.orchestra.run.smithi006.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 1.0 MB/s | 185 kB 00:00 2023-09-30T00:41:26.518 INFO:teuthology.orchestra.run.smithi006.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-09-30T00:41:26.519 INFO:teuthology.orchestra.run.smithi022.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 45 kB/s | 44 kB 00:00 2023-09-30T00:41:26.544 INFO:teuthology.orchestra.run.smithi006.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-09-30T00:41:26.569 INFO:teuthology.orchestra.run.smithi006.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 974 kB/s | 49 kB 00:00 2023-09-30T00:41:26.632 INFO:teuthology.orchestra.run.smithi006.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 670 kB/s | 59 kB 00:00 2023-09-30T00:41:26.669 INFO:teuthology.orchestra.run.smithi006.stdout:(22/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.8 MB/s | 283 kB 00:00 2023-09-30T00:41:26.695 INFO:teuthology.orchestra.run.smithi006.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 248 kB/s | 16 kB 00:00 2023-09-30T00:41:26.720 INFO:teuthology.orchestra.run.smithi006.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 372 kB/s | 19 kB 00:00 2023-09-30T00:41:26.746 INFO:teuthology.orchestra.run.smithi006.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-09-30T00:41:26.771 INFO:teuthology.orchestra.run.smithi006.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 584 kB/s | 29 kB 00:00 2023-09-30T00:41:26.813 INFO:teuthology.orchestra.run.smithi006.stdout:(27/32): python3-waitress-1.4.3-1.el8.noarch.rp 6.0 MB/s | 255 kB 00:00 2023-09-30T00:41:26.855 INFO:teuthology.orchestra.run.smithi006.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 253 kB/s | 27 kB 00:00 2023-09-30T00:41:26.880 INFO:teuthology.orchestra.run.smithi006.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.7 MB/s | 251 kB 00:00 2023-09-30T00:41:26.922 INFO:teuthology.orchestra.run.smithi006.stdout:(30/32): python3-zc-lockfile-2.0-2.el8.noarch.r 552 kB/s | 23 kB 00:00 2023-09-30T00:41:26.977 INFO:teuthology.orchestra.run.smithi022.stdout:(14/32): ceph-mon-17.2.6-1165.g93dce5e1.el8.x86 908 kB/s | 3.9 MB 00:04 2023-09-30T00:41:27.089 INFO:teuthology.orchestra.run.smithi006.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 365 kB/s | 85 kB 00:00 2023-09-30T00:41:27.244 INFO:teuthology.orchestra.run.smithi022.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 255 kB/s | 185 kB 00:00 2023-09-30T00:41:27.802 INFO:teuthology.orchestra.run.smithi022.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 689 kB/s | 384 kB 00:00 2023-09-30T00:41:28.036 INFO:teuthology.orchestra.run.smithi022.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 47 kB/s | 11 kB 00:00 2023-09-30T00:41:28.278 INFO:teuthology.orchestra.run.smithi022.stdout:(18/32): python3-jaraco-functools-2.0-4.el8.noa 73 kB/s | 18 kB 00:00 2023-09-30T00:41:28.304 INFO:teuthology.orchestra.run.smithi022.stdout:(19/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 130 kB/s | 173 kB 00:01 2023-09-30T00:41:28.520 INFO:teuthology.orchestra.run.smithi022.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 201 kB/s | 49 kB 00:00 2023-09-30T00:41:28.571 INFO:teuthology.orchestra.run.smithi022.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 220 kB/s | 59 kB 00:00 2023-09-30T00:41:28.813 INFO:teuthology.orchestra.run.smithi022.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 64 kB/s | 16 kB 00:00 2023-09-30T00:41:28.838 INFO:teuthology.orchestra.run.smithi022.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 892 kB/s | 283 kB 00:00 2023-09-30T00:41:29.055 INFO:teuthology.orchestra.run.smithi022.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 77 kB/s | 19 kB 00:00 2023-09-30T00:41:29.089 INFO:teuthology.orchestra.run.smithi022.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 96 kB/s | 24 kB 00:00 2023-09-30T00:41:29.306 INFO:teuthology.orchestra.run.smithi022.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 117 kB/s | 29 kB 00:00 2023-09-30T00:41:29.331 INFO:teuthology.orchestra.run.smithi022.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 114 kB/s | 27 kB 00:00 2023-09-30T00:41:29.639 INFO:teuthology.orchestra.run.smithi022.stdout:(28/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 815 kB/s | 251 kB 00:00 2023-09-30T00:41:29.732 INFO:teuthology.orchestra.run.smithi022.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 600 kB/s | 255 kB 00:00 2023-09-30T00:41:29.907 INFO:teuthology.orchestra.run.smithi022.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 320 kB/s | 85 kB 00:00 2023-09-30T00:41:29.957 INFO:teuthology.orchestra.run.smithi022.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 102 kB/s | 23 kB 00:00 2023-09-30T00:41:30.889 INFO:teuthology.orchestra.run.smithi006.stdout:(32/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 2.3 MB/s | 17 MB 00:07 2023-09-30T00:41:30.892 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:30.893 INFO:teuthology.orchestra.run.smithi006.stdout:Total 3.1 MB/s | 28 MB 00:09 2023-09-30T00:41:30.984 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:41:31.025 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:41:31.026 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:41:31.291 INFO:teuthology.orchestra.run.smithi022.stdout:(32/32): ceph-osd-17.2.6-1165.g93dce5e1.el8.x86 2.2 MB/s | 17 MB 00:07 2023-09-30T00:41:31.295 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:31.295 INFO:teuthology.orchestra.run.smithi022.stdout:Total 3.1 MB/s | 28 MB 00:09 2023-09-30T00:41:31.353 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:41:31.356 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:41:31.387 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:41:31.428 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:41:31.428 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:41:31.761 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:41:31.764 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:41:32.481 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:41:32.719 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T00:41:32.891 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:41:32.914 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T00:41:33.105 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-09-30T00:41:33.126 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T00:41:33.326 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-09-30T00:41:33.379 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T00:41:33.526 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T00:41:33.547 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-09-30T00:41:33.766 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-09-30T00:41:33.843 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T00:41:33.963 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-09-30T00:41:34.024 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T00:41:34.173 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-09-30T00:41:34.261 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T00:41:34.344 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-09-30T00:41:34.482 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T00:41:34.523 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-09-30T00:41:34.701 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T00:41:34.773 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-09-30T00:41:34.949 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T00:41:34.974 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-09-30T00:41:35.207 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T00:41:35.247 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-09-30T00:41:35.481 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-09-30T00:41:35.599 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T00:41:35.828 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T00:41:35.940 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-09-30T00:41:36.053 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T00:41:36.225 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-09-30T00:41:36.282 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T00:41:36.431 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-09-30T00:41:36.607 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:41:36.699 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-09-30T00:41:36.877 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:41:36.952 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T00:41:37.145 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T00:41:37.207 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-09-30T00:41:37.367 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T00:41:37.400 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-09-30T00:41:37.618 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T00:41:37.638 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-09-30T00:41:37.813 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-09-30T00:41:37.891 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T00:41:38.099 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-09-30T00:41:38.135 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T00:41:38.314 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-09-30T00:41:38.407 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T00:41:38.542 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-09-30T00:41:38.859 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T00:41:38.887 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-09-30T00:41:39.200 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T00:41:39.261 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-09-30T00:41:39.574 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T00:41:39.630 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 27/32 2023-09-30T00:41:39.795 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:41:39.842 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:41:41.440 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:41:41.492 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 28/32 2023-09-30T00:41:41.628 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:41:41.666 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:41:42.066 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:41:42.105 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 29/32 2023-09-30T00:41:42.237 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:41:42.272 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:41:42.477 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:41:42.513 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 30/32 2023-09-30T00:41:42.647 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:41:42.653 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:41:42.669 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:41:42.676 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 31/32 2023-09-30T00:41:42.764 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:41:42.812 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:41:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:41:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T00:41:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T00:41:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T00:41:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T00:41:43.218 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T00:41:43.219 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T00:41:43.220 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T00:41:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T00:41:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T00:41:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T00:41:43.221 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 32/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 3/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 4/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 5/32 2023-09-30T00:41:43.294 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8. 6/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-09-30T00:41:43.295 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-09-30T00:41:43.296 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-09-30T00:41:43.297 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-09-30T00:41:43.297 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-09-30T00:41:43.297 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-09-30T00:41:43.297 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-09-30T00:41:43.297 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-09-30T00:41:43.297 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-09-30T00:41:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T00:41:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:44.378 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:41:44.378 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T00:41:44.379 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T00:41:44.380 INFO:teuthology.orchestra.run.smithi022.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:44.381 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:41:44.438 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-09-30T00:41:44.438 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:44.438 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:41:44.438 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.438 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mds-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.438 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-modules-core-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mon-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-osd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-6.2-6.el8.noarch 2023-09-30T00:41:44.439 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-mako-1.0.6-14.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-portend-2.6-1.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pycparser-2.14-14.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pytz-2017.2-11.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-09-30T00:41:44.440 INFO:teuthology.orchestra.run.smithi006.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:44.441 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:41:44.546 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-base 2023-09-30T00:41:44.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-base 2023-09-30T00:41:45.530 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:40 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:41:45.578 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:41 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:41:47.072 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:41:47.130 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-base-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:41:47.282 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:41:47.283 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:41:47.283 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:41:47.343 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:41:47.344 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:41:47.345 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:41:47.522 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install cephadm 2023-09-30T00:41:47.957 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install cephadm 2023-09-30T00:41:48.504 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:44 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:41:48.937 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:43 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:41:50.256 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:50.257 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:41:50.257 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:50.257 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:41:50.257 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:50.257 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 80 k 2023-09-30T00:41:50.257 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 363 k 2023-09-30T00:41:50.258 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:41:50.653 INFO:teuthology.orchestra.run.smithi006.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 203 kB/s | 80 kB 00:00 2023-09-30T00:41:50.653 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:50.654 INFO:teuthology.orchestra.run.smithi006.stdout:Total 202 kB/s | 80 kB 00:00 2023-09-30T00:41:50.654 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:41:50.664 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:41:50.664 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:41:50.666 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:41:50.670 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:41:50.696 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 80 k 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:41:50.697 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:50.698 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:41:50.698 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:50.698 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 80 k 2023-09-30T00:41:50.698 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 363 k 2023-09-30T00:41:50.698 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:41:50.715 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:41:51.096 INFO:teuthology.orchestra.run.smithi022.stdout:cephadm-17.2.6-1165.g93dce5e1.el8.noarch.rpm 201 kB/s | 80 kB 00:00 2023-09-30T00:41:51.097 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:51.097 INFO:teuthology.orchestra.run.smithi022.stdout:Total 201 kB/s | 80 kB 00:00 2023-09-30T00:41:51.097 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:41:51.107 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:41:51.107 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:41:51.110 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:41:51.114 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:41:51.163 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:41:51.246 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:51.382 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:51.642 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:51.671 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:51.811 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:52.086 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:52.302 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:52.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:52.303 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:41:52.303 INFO:teuthology.orchestra.run.smithi006.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:41:52.303 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:52.303 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:41:52.525 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:41:52.746 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:41:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:52.747 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:41:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:41:52.747 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:52.747 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:41:52.932 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-immutable-object-cache 2023-09-30T00:41:53.518 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:49 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:41:53.925 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:48 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:41:55.274 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:41:55.274 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:41:55.275 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:55.276 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 161 k 2023-09-30T00:41:55.276 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 490 k 2023-09-30T00:41:55.276 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:41:55.677 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:41:55.679 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:55.679 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2023-09-30T00:41:55.679 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:55.679 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:41:55.680 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 161 k 2023-09-30T00:41:55.680 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:55.680 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:41:55.680 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:41:55.680 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:41:55.681 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:55.681 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 161 k 2023-09-30T00:41:55.681 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 490 k 2023-09-30T00:41:55.681 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:41:55.973 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 232 kB/s | 161 kB 00:00 2023-09-30T00:41:55.974 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:55.974 INFO:teuthology.orchestra.run.smithi006.stdout:Total 231 kB/s | 161 kB 00:00 2023-09-30T00:41:55.975 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:41:55.989 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:41:55.989 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:41:56.047 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:41:56.050 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:41:56.170 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:41:56.344 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:41:56.400 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-immutable-object-cache-17.2.6-1165.g93dce5 224 kB/s | 161 kB 00:00 2023-09-30T00:41:56.400 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:41:56.401 INFO:teuthology.orchestra.run.smithi022.stdout:Total 224 kB/s | 161 kB 00:00 2023-09-30T00:41:56.401 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:41:56.415 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:41:56.415 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:41:56.473 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:41:56.476 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:41:56.645 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:41:56.804 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:41:56.804 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:41:57.276 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:41:57.498 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:41:57.498 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:57.498 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:41:57.498 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:57.498 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:41:57.498 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:41:57.783 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr 2023-09-30T00:41:58.000 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1. 1/1 2023-09-30T00:41:58.000 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:58.000 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:41:58.001 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-immutable-object-cache-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:41:58.001 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:41:58.001 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:41:58.246 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr 2023-09-30T00:41:58.764 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:41:59.251 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:54 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:00.310 INFO:teuthology.orchestra.run.smithi006.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:42:00.519 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:00.521 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:42:00.521 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:00.634 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:42:00.806 INFO:teuthology.orchestra.run.smithi022.stdout:Package ceph-mgr-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:42:01.023 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:01.024 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:42:01.025 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:01.131 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-dashboard 2023-09-30T00:42:01.615 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:02.114 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:01:57 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:03.380 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:42:03.381 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout:Install 6 Packages 2023-09-30T00:42:03.382 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:03.383 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 2.0 M 2023-09-30T00:42:03.383 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 8.0 M 2023-09-30T00:42:03.383 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:03.759 INFO:teuthology.orchestra.run.smithi006.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 87 kB/s | 23 kB 00:00 2023-09-30T00:42:03.785 INFO:teuthology.orchestra.run.smithi006.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 50 kB/s | 14 kB 00:00 2023-09-30T00:42:03.860 INFO:teuthology.orchestra.run.smithi006.stdout:(3/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 450 kB/s | 34 kB 00:00 2023-09-30T00:42:03.863 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:03.864 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:03.865 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:42:03.865 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:03.865 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:03.865 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 1.7 M 2023-09-30T00:42:03.865 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 23 k 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 14 k 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:03.866 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:03.867 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:03.867 INFO:teuthology.orchestra.run.smithi022.stdout:Install 6 Packages 2023-09-30T00:42:03.867 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:03.868 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 2.0 M 2023-09-30T00:42:03.868 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 8.0 M 2023-09-30T00:42:03.868 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:03.952 INFO:teuthology.orchestra.run.smithi006.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 222 kB/s | 43 kB 00:00 2023-09-30T00:42:03.994 INFO:teuthology.orchestra.run.smithi006.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2023-09-30T00:42:04.211 INFO:teuthology.orchestra.run.smithi006.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 2.4 MB/s | 1.7 MB 00:00 2023-09-30T00:42:04.213 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:04.213 INFO:teuthology.orchestra.run.smithi006.stdout:Total 2.4 MB/s | 2.0 MB 00:00 2023-09-30T00:42:04.227 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:04.238 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:04.238 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:04.333 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:04.337 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:04.607 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:04.678 INFO:teuthology.orchestra.run.smithi022.stdout:(1/6): ceph-grafana-dashboards-17.2.6-1165.g93d 83 kB/s | 23 kB 00:00 2023-09-30T00:42:04.753 INFO:teuthology.orchestra.run.smithi022.stdout:(2/6): ceph-prometheus-alerts-17.2.6-1165.g93dc 41 kB/s | 14 kB 00:00 2023-09-30T00:42:04.821 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T00:42:04.846 INFO:teuthology.orchestra.run.smithi022.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 255 kB/s | 43 kB 00:00 2023-09-30T00:42:04.871 INFO:teuthology.orchestra.run.smithi022.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 287 kB/s | 34 kB 00:00 2023-09-30T00:42:04.988 INFO:teuthology.orchestra.run.smithi022.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.4 MB/s | 196 kB 00:00 2023-09-30T00:42:05.039 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T00:42:05.254 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T00:42:05.321 INFO:teuthology.orchestra.run.smithi022.stdout:(6/6): ceph-mgr-dashboard-17.2.6-1165.g93dce5e1 1.8 MB/s | 1.7 MB 00:00 2023-09-30T00:42:05.323 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:05.323 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2023-09-30T00:42:05.336 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:05.347 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:05.347 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:05.366 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:42:05.441 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:05.445 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:05.720 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:42:05.773 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:05.963 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-09-30T00:42:05.991 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:42:06.182 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-09-30T00:42:06.310 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:42:06.310 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:42:06.310 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:42:06.310 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:42:06.310 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T00:42:06.310 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T00:42:06.346 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-09-30T00:42:06.484 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 4/6 2023-09-30T00:42:06.829 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 5/6 2023-09-30T00:42:07.010 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T00:42:07.010 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:07.011 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:07.045 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:42:07.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:42:07.369 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 6/6 2023-09-30T00:42:07.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8. 1/6 2023-09-30T00:42:07.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarc 2/6 2023-09-30T00:42:07.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.n 3/6 2023-09-30T00:42:07.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-09-30T00:42:07.369 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-09-30T00:42:08.093 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-09-30T00:42:08.093 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:08.093 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-grafana-dashboards-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-dashboard-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-prometheus-alerts-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: python3-routes-2.4.1-12.el8.noarch 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:08.094 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:08.134 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:04 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:08.239 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-diskprediction-local 2023-09-30T00:42:09.221 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:04 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:09.895 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:09.896 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================================ 2023-09-30T00:42:09.896 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:42:09.896 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================================ 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T00:42:09.897 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T00:42:09.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:09.898 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:09.898 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================================ 2023-09-30T00:42:09.898 INFO:teuthology.orchestra.run.smithi006.stdout:Install 8 Packages 2023-09-30T00:42:09.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:09.899 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 35 M 2023-09-30T00:42:09.899 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 210 M 2023-09-30T00:42:09.899 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:10.984 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================================ 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================================ 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 7.4 M 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran x86_64 8.5.0-20.el8 baseos 645 k 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath x86_64 8.5.0-20.el8 baseos 172 k 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-09-30T00:42:10.986 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================================ 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout:Install 8 Packages 2023-09-30T00:42:10.987 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:10.988 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 35 M 2023-09-30T00:42:10.988 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 210 M 2023-09-30T00:42:10.988 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:12.526 INFO:teuthology.orchestra.run.smithi006.stdout:(1/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2023-09-30T00:42:12.626 INFO:teuthology.orchestra.run.smithi006.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 8.9 MB/s | 4.7 MB 00:00 2023-09-30T00:42:12.702 INFO:teuthology.orchestra.run.smithi006.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.9 MB/s | 225 kB 00:00 2023-09-30T00:42:12.777 INFO:teuthology.orchestra.run.smithi006.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2023-09-30T00:42:12.795 INFO:teuthology.orchestra.run.smithi022.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 9.9 MB/s | 4.7 MB 00:00 2023-09-30T00:42:12.870 INFO:teuthology.orchestra.run.smithi022.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.8 MB/s | 4.9 MB 00:00 2023-09-30T00:42:12.929 INFO:teuthology.orchestra.run.smithi022.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2023-09-30T00:42:13.054 INFO:teuthology.orchestra.run.smithi022.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 14 MB/s | 3.7 MB 00:00 2023-09-30T00:42:13.077 INFO:teuthology.orchestra.run.smithi006.stdout:(5/8): libgfortran-8.5.0-20.el8.x86_64.rpm 2.1 MB/s | 645 kB 00:00 2023-09-30T00:42:13.413 INFO:teuthology.orchestra.run.smithi022.stdout:(5/8): python3-scipy-1.0.0-21.module_el8.5.0+77 28 MB/s | 14 MB 00:00 2023-09-30T00:42:13.480 INFO:teuthology.orchestra.run.smithi022.stdout:(6/8): libgfortran-8.5.0-20.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2023-09-30T00:42:13.512 INFO:teuthology.orchestra.run.smithi006.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 17 MB/s | 14 MB 00:00 2023-09-30T00:42:13.647 INFO:teuthology.orchestra.run.smithi022.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 740 kB/s | 172 kB 00:00 2023-09-30T00:42:13.929 INFO:teuthology.orchestra.run.smithi006.stdout:(7/8): libquadmath-8.5.0-20.el8.x86_64.rpm 202 kB/s | 172 kB 00:00 2023-09-30T00:42:15.656 INFO:teuthology.orchestra.run.smithi022.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 2.2 MB/s | 7.4 MB 00:03 2023-09-30T00:42:15.656 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:15.657 INFO:teuthology.orchestra.run.smithi022.stdout:Total 7.6 MB/s | 35 MB 00:04 2023-09-30T00:42:15.910 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:15.931 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:15.931 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:16.171 INFO:teuthology.orchestra.run.smithi006.stdout:(8/8): ceph-mgr-diskprediction-local-17.2.6-116 1.8 MB/s | 7.4 MB 00:04 2023-09-30T00:42:16.172 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:16.172 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.6 MB/s | 35 MB 00:06 2023-09-30T00:42:16.350 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:16.354 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:16.426 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:16.447 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:16.447 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:16.896 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:16.900 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:16.963 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:17.158 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:42:17.309 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:42:17.441 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:42:17.495 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:17.724 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:42:17.864 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libquadmath-8.5.0-20.el8.x86_64 1/8 2023-09-30T00:42:18.006 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:42:18.101 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:42:18.332 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:42:18.675 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libgfortran-8.5.0-20.el8.x86_64 2/8 2023-09-30T00:42:18.865 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:42:19.020 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:42:19.183 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:42:19.551 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:42:19.707 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:42:19.965 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:42:20.306 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:42:20.487 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-09-30T00:42:20.557 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:42:20.832 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:42:21.098 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:42:22.565 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:42:23.086 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-09-30T00:42:23.948 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T00:42:24.140 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:42:24.402 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T00:42:24.497 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T00:42:24.614 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:42:24.980 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 8/8 2023-09-30T00:42:24.980 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e 1/8 2023-09-30T00:42:24.981 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-09-30T00:42:24.981 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-09-30T00:42:24.981 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-09-30T00:42:24.981 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-09-30T00:42:24.981 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-09-30T00:42:24.981 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libgfortran-8.5.0-20.el8.x86_64 7/8 2023-09-30T00:42:25.224 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T00:42:25.224 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:25.225 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:25.408 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:42:25.691 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libquadmath-8.5.0-20.el8.x86_64 8/8 2023-09-30T00:42:25.691 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:25.691 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:42:25.691 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-diskprediction-local-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:25.691 INFO:teuthology.orchestra.run.smithi006.stdout: libgfortran-8.5.0-20.el8.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: libquadmath-8.5.0-20.el8.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-0.3.15-4.el8.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:25.692 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:25.831 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-rook 2023-09-30T00:42:26.409 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:21 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:26.820 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:22 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:28.169 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:28.171 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:28.171 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:42:28.171 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:28.171 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:28.171 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:42:28.171 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:42:28.172 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:42:28.173 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:28.173 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:28.173 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:28.173 INFO:teuthology.orchestra.run.smithi022.stdout:Install 13 Packages 2023-09-30T00:42:28.173 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:28.173 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 1.9 M 2023-09-30T00:42:28.174 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 25 M 2023-09-30T00:42:28.174 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:28.582 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:28.584 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 47 k 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-09-30T00:42:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:28.586 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:28.587 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:28.587 INFO:teuthology.orchestra.run.smithi006.stdout:Install 13 Packages 2023-09-30T00:42:28.587 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:28.587 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 1.9 M 2023-09-30T00:42:28.587 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 25 M 2023-09-30T00:42:28.588 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:28.862 INFO:teuthology.orchestra.run.smithi022.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 186 kB/s | 27 kB 00:00 2023-09-30T00:42:28.946 INFO:teuthology.orchestra.run.smithi022.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 125 kB/s | 20 kB 00:00 2023-09-30T00:42:28.946 INFO:teuthology.orchestra.run.smithi006.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 147 kB/s | 20 kB 00:00 2023-09-30T00:42:28.946 INFO:teuthology.orchestra.run.smithi006.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 164 kB/s | 27 kB 00:00 2023-09-30T00:42:28.986 INFO:teuthology.orchestra.run.smithi006.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-09-30T00:42:28.995 INFO:teuthology.orchestra.run.smithi022.stdout:(3/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.1 MB/s | 126 kB 00:00 2023-09-30T00:42:29.011 INFO:teuthology.orchestra.run.smithi006.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 940 kB/s | 110 kB 00:00 2023-09-30T00:42:29.021 INFO:teuthology.orchestra.run.smithi022.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 779 kB/s | 110 kB 00:00 2023-09-30T00:42:29.112 INFO:teuthology.orchestra.run.smithi006.stdout:(5/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 125 kB/s | 47 kB 00:00 2023-09-30T00:42:29.163 INFO:teuthology.orchestra.run.smithi006.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 283 kB/s | 43 kB 00:00 2023-09-30T00:42:29.187 INFO:teuthology.orchestra.run.smithi022.stdout:(5/13): python3-requests-oauthlib-1.0.0-1.el8.n 256 kB/s | 43 kB 00:00 2023-09-30T00:42:29.196 INFO:teuthology.orchestra.run.smithi006.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 739 kB/s | 155 kB 00:00 2023-09-30T00:42:29.213 INFO:teuthology.orchestra.run.smithi022.stdout:(6/13): ceph-mgr-rook-17.2.6-1165.g93dce5e1.el8 96 kB/s | 47 kB 00:00 2023-09-30T00:42:29.247 INFO:teuthology.orchestra.run.smithi022.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 617 kB/s | 155 kB 00:00 2023-09-30T00:42:29.764 INFO:teuthology.orchestra.run.smithi022.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 30 kB/s | 16 kB 00:00 2023-09-30T00:42:29.789 INFO:teuthology.orchestra.run.smithi022.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 55 kB/s | 33 kB 00:00 2023-09-30T00:42:29.796 INFO:teuthology.orchestra.run.smithi006.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 48 kB/s | 33 kB 00:00 2023-09-30T00:42:29.813 INFO:teuthology.orchestra.run.smithi006.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 25 kB/s | 16 kB 00:00 2023-09-30T00:42:29.839 INFO:teuthology.orchestra.run.smithi006.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 127 kB/s | 81 kB 00:00 2023-09-30T00:42:29.931 INFO:teuthology.orchestra.run.smithi022.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 119 kB/s | 81 kB 00:00 2023-09-30T00:42:29.939 INFO:teuthology.orchestra.run.smithi006.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 613 kB/s | 61 kB 00:00 2023-09-30T00:42:30.081 INFO:teuthology.orchestra.run.smithi006.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 240 kB/s | 64 kB 00:00 2023-09-30T00:42:30.323 INFO:teuthology.orchestra.run.smithi022.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 120 kB/s | 64 kB 00:00 2023-09-30T00:42:30.456 INFO:teuthology.orchestra.run.smithi022.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 117 kB/s | 61 kB 00:00 2023-09-30T00:42:31.290 INFO:teuthology.orchestra.run.smithi022.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 782 kB/s | 1.2 MB 00:01 2023-09-30T00:42:31.291 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:31.291 INFO:teuthology.orchestra.run.smithi022.stdout:Total 634 kB/s | 1.9 MB 00:03 2023-09-30T00:42:31.358 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:31.376 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:31.376 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:31.439 INFO:teuthology.orchestra.run.smithi006.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 727 kB/s | 1.2 MB 00:01 2023-09-30T00:42:31.442 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:31.442 INFO:teuthology.orchestra.run.smithi006.stdout:Total 692 kB/s | 1.9 MB 00:02 2023-09-30T00:42:31.475 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:31.478 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:31.507 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:31.525 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:31.525 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:31.623 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:31.626 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:32.022 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:32.199 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:32.274 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:42:32.432 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-09-30T00:42:32.471 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T00:42:32.636 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-09-30T00:42:32.730 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T00:42:32.879 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-09-30T00:42:32.939 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T00:42:33.080 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-09-30T00:42:33.148 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T00:42:33.256 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-09-30T00:42:33.376 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T00:42:33.474 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-09-30T00:42:33.697 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T00:42:33.771 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-09-30T00:42:33.940 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:42:34.006 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-09-30T00:42:35.115 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:42:35.150 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-09-30T00:42:35.396 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T00:42:35.421 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-09-30T00:42:35.597 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:42:35.614 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-09-30T00:42:35.783 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T00:42:35.803 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-09-30T00:42:35.946 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:42:36.300 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:42:36.300 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:42:36.300 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T00:42:36.300 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T00:42:36.300 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T00:42:36.301 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T00:42:36.371 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:42:36.740 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 13/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 1/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-09-30T00:42:36.741 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-09-30T00:42:36.742 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-09-30T00:42:36.742 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-09-30T00:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T00:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:42:37.052 INFO:teuthology.orchestra.run.smithi006.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:37.053 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:37.176 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:42:37.516 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-09-30T00:42:37.516 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-rook-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-09-30T00:42:37.517 INFO:teuthology.orchestra.run.smithi022.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: python3-rsa-4.9-2.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:37.518 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:37.662 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-mgr-cephadm 2023-09-30T00:42:38.159 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:34 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:38.645 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:33 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:39.924 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:39.925 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:39.925 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:39.926 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:39.927 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:39.927 INFO:teuthology.orchestra.run.smithi006.stdout:Install 5 Packages 2023-09-30T00:42:39.927 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:39.927 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 6.0 M 2023-09-30T00:42:39.927 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 26 M 2023-09-30T00:42:39.928 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:40.401 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 121 k 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh noarch 2.7.0-2.el8 epel 476 k 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-09-30T00:42:40.402 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-09-30T00:42:40.403 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort noarch 7.1.1-2.el8 epel 60 k 2023-09-30T00:42:40.403 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:40.403 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:40.403 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:40.403 INFO:teuthology.orchestra.run.smithi022.stdout:Install 5 Packages 2023-09-30T00:42:40.403 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:40.404 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 6.0 M 2023-09-30T00:42:40.404 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 26 M 2023-09-30T00:42:40.404 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:40.655 INFO:teuthology.orchestra.run.smithi006.stdout:(1/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 634 kB/s | 121 kB 00:00 2023-09-30T00:42:40.714 INFO:teuthology.orchestra.run.smithi006.stdout:(2/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 2.1 MB/s | 538 kB 00:00 2023-09-30T00:42:40.872 INFO:teuthology.orchestra.run.smithi006.stdout:(3/5): python3-natsort-7.1.1-2.el8.noarch.rpm 381 kB/s | 60 kB 00:00 2023-09-30T00:42:40.956 INFO:teuthology.orchestra.run.smithi006.stdout:(4/5): python3-babel-2.5.1-7.el8.noarch.rpm 9.7 MB/s | 4.8 MB 00:00 2023-09-30T00:42:41.022 INFO:teuthology.orchestra.run.smithi022.stdout:(1/5): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2023-09-30T00:42:41.023 INFO:teuthology.orchestra.run.smithi006.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 1.3 MB/s | 476 kB 00:00 2023-09-30T00:42:41.023 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:41.024 INFO:teuthology.orchestra.run.smithi006.stdout:Total 5.4 MB/s | 6.0 MB 00:01 2023-09-30T00:42:41.056 INFO:teuthology.orchestra.run.smithi022.stdout:(2/5): ceph-mgr-cephadm-17.2.6-1165.g93dce5e1.e 398 kB/s | 121 kB 00:00 2023-09-30T00:42:41.082 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:41.095 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:41.095 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:41.208 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:41.212 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:41.302 INFO:teuthology.orchestra.run.smithi022.stdout:(3/5): python3-babel-2.5.1-7.el8.noarch.rpm 8.7 MB/s | 4.8 MB 00:00 2023-09-30T00:42:41.521 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:41.548 INFO:teuthology.orchestra.run.smithi022.stdout:(4/5): python3-natsort-7.1.1-2.el8.noarch.rpm 122 kB/s | 60 kB 00:00 2023-09-30T00:42:41.573 INFO:teuthology.orchestra.run.smithi022.stdout:(5/5): python3-asyncssh-2.7.0-2.el8.noarch.rpm 864 kB/s | 476 kB 00:00 2023-09-30T00:42:41.574 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:41.574 INFO:teuthology.orchestra.run.smithi022.stdout:Total 5.1 MB/s | 6.0 MB 00:01 2023-09-30T00:42:41.632 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:41.645 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:41.645 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:41.756 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:41.760 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:41.846 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T00:42:42.101 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:42.461 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-natsort-7.1.1-2.el8.noarch 1/5 2023-09-30T00:42:42.864 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:42:43.241 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T00:42:43.455 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-asyncssh-2.7.0-2.el8.noarch 2/5 2023-09-30T00:42:43.537 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T00:42:43.720 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:42:43.791 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 3/5 2023-09-30T00:42:44.035 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 4/5 2023-09-30T00:42:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:42:44.048 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:42:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T00:42:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T00:42:44.049 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:42:44.253 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:42:44.585 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 5/5 2023-09-30T00:42:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 1/5 2023-09-30T00:42:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/5 2023-09-30T00:42:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/5 2023-09-30T00:42:44.586 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-asyncssh-2.7.0-2.el8.noarch 4/5 2023-09-30T00:42:44.741 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:44.742 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:44.873 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-fuse 2023-09-30T00:42:45.288 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : python3-natsort-7.1.1-2.el8.noarch 5/5 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-mgr-cephadm-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-asyncssh-2.7.0-2.el8.noarch 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-babel-2.5.1-7.el8.noarch 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: python3-natsort-7.1.1-2.el8.noarch 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:45.289 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:45.415 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-fuse 2023-09-30T00:42:45.865 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:46.408 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:41 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:47.626 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout:Installing dependencies: 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T00:42:47.627 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:47.628 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:47.628 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:47.628 INFO:teuthology.orchestra.run.smithi006.stdout:Install 2 Packages 2023-09-30T00:42:47.628 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:47.628 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 957 k 2023-09-30T00:42:47.628 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 3.0 M 2023-09-30T00:42:47.629 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:48.014 INFO:teuthology.orchestra.run.smithi006.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 414 kB/s | 83 kB 00:00 2023-09-30T00:42:48.171 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 874 k 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout:Installing dependencies: 2023-09-30T00:42:48.172 INFO:teuthology.orchestra.run.smithi022.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout:Install 2 Packages 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 957 k 2023-09-30T00:42:48.173 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 3.0 M 2023-09-30T00:42:48.174 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:48.331 INFO:teuthology.orchestra.run.smithi006.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 1.7 MB/s | 874 kB 00:00 2023-09-30T00:42:48.332 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:48.332 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.3 MB/s | 957 kB 00:00 2023-09-30T00:42:48.346 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:48.370 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:48.370 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:48.473 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:48.477 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:48.566 INFO:teuthology.orchestra.run.smithi022.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 415 kB/s | 83 kB 00:00 2023-09-30T00:42:48.667 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:48.774 INFO:teuthology.orchestra.run.smithi022.stdout:(2/2): ceph-fuse-17.2.6-1165.g93dce5e1.el8.x86_ 2.1 MB/s | 874 kB 00:00 2023-09-30T00:42:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:48.775 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.6 MB/s | 957 kB 00:00 2023-09-30T00:42:48.788 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:48.813 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:48.813 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:48.915 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:48.918 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:48.937 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T00:42:49.138 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:49.154 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:42:49.393 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-09-30T00:42:49.581 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:42:49.581 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:42:49.604 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:42:50.064 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2/2 2023-09-30T00:42:50.065 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/2 2023-09-30T00:42:50.279 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T00:42:50.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:50.279 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:42:50.279 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T00:42:50.279 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:50.279 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:50.780 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-09-30T00:42:50.780 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:50.780 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:42:50.780 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-09-30T00:42:50.780 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:50.780 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:51.058 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install ceph-volume 2023-09-30T00:42:51.220 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install ceph-volume 2023-09-30T00:42:52.060 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:47 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:52.223 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:48 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:53.826 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repository Size 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:42:53.827 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:53.828 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 267 k 2023-09-30T00:42:53.828 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 1.3 M 2023-09-30T00:42:53.828 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:53.979 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repository Size 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume noarch 2:17.2.6-1165.g93dce5e1.el8 ceph-noarch 267 k 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:42:53.981 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:53.982 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 267 k 2023-09-30T00:42:53.982 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 1.3 M 2023-09-30T00:42:53.982 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:54.039 INFO:teuthology.orchestra.run.smithi022.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.2 MB/s | 267 kB 00:00 2023-09-30T00:42:54.040 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:54.040 INFO:teuthology.orchestra.run.smithi022.stdout:Total 1.2 MB/s | 267 kB 00:00 2023-09-30T00:42:54.041 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:54.053 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:54.053 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:54.095 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:54.098 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:54.205 INFO:teuthology.orchestra.run.smithi006.stdout:ceph-volume-17.2.6-1165.g93dce5e1.el8.noarch.rp 1.2 MB/s | 267 kB 00:00 2023-09-30T00:42:54.206 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:54.206 INFO:teuthology.orchestra.run.smithi006.stdout:Total 1.2 MB/s | 267 kB 00:00 2023-09-30T00:42:54.206 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:54.219 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:54.219 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:54.260 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:54.263 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:54.304 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:54.456 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:54.552 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:42:54.678 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:42:55.025 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:42:55.159 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:42:55.778 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:42:55.778 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:55.778 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:42:55.778 INFO:teuthology.orchestra.run.smithi022.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:55.778 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:55.778 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:42:55.850 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 1/1 2023-09-30T00:42:55.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:55.850 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:42:55.850 INFO:teuthology.orchestra.run.smithi006.stdout: ceph-volume-2:17.2.6-1165.g93dce5e1.el8.noarch 2023-09-30T00:42:55.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:55.850 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:42:56.021 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados-devel 2023-09-30T00:42:56.071 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados-devel 2023-09-30T00:42:57.023 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:52 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:42:57.065 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:53 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:42:58.803 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:42:58.804 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:42:58.805 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 130 k 2023-09-30T00:42:58.805 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 464 k 2023-09-30T00:42:58.805 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:42:58.849 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:42:58.849 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:58.849 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 130 k 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:42:58.850 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:42:58.851 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 130 k 2023-09-30T00:42:58.851 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 464 k 2023-09-30T00:42:58.851 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:42:59.343 INFO:teuthology.orchestra.run.smithi022.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 241 kB/s | 130 kB 00:00 2023-09-30T00:42:59.344 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:59.344 INFO:teuthology.orchestra.run.smithi022.stdout:Total 241 kB/s | 130 kB 00:00 2023-09-30T00:42:59.344 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:42:59.351 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:42:59.351 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:42:59.412 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:42:59.416 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:42:59.496 INFO:teuthology.orchestra.run.smithi006.stdout:librados-devel-17.2.6-1165.g93dce5e1.el8.x86_64 202 kB/s | 130 kB 00:00 2023-09-30T00:42:59.496 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:42:59.496 INFO:teuthology.orchestra.run.smithi006.stdout:Total 201 kB/s | 130 kB 00:00 2023-09-30T00:42:59.497 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:42:59.503 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:42:59.504 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:42:59.537 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:42:59.565 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:42:59.569 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:42:59.691 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:42:59.818 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:42:59.935 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:00.168 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:00.261 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:00.895 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:43:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:00.895 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:00.895 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:00.932 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:00.932 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:00.932 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:43:00.932 INFO:teuthology.orchestra.run.smithi006.stdout: librados-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:00.933 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:00.933 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:01.127 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs2 2023-09-30T00:43:01.160 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs2 2023-09-30T00:43:02.133 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:02:57 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:02.154 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:02:58 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:03.698 INFO:teuthology.orchestra.run.smithi022.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:03.703 INFO:teuthology.orchestra.run.smithi006.stdout:Package libcephfs2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:03.911 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:03.912 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:03.912 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:03.913 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:03.913 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:03.913 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:04.017 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install libcephfs-devel 2023-09-30T00:43:04.020 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install libcephfs-devel 2023-09-30T00:43:05.010 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:01 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:05.012 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:00 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:06.764 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:06.765 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:43:06.766 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:06.766 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 25 k 2023-09-30T00:43:06.766 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 109 k 2023-09-30T00:43:06.766 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:43:06.775 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:06.775 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:06.775 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 25 k 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:43:06.776 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:06.777 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 25 k 2023-09-30T00:43:06.777 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 109 k 2023-09-30T00:43:06.777 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:43:07.066 INFO:teuthology.orchestra.run.smithi006.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 86 kB/s | 25 kB 00:00 2023-09-30T00:43:07.066 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:07.066 INFO:teuthology.orchestra.run.smithi006.stdout:Total 86 kB/s | 25 kB 00:00 2023-09-30T00:43:07.067 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:43:07.070 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:43:07.070 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:43:07.073 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:43:07.076 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:43:07.122 INFO:teuthology.orchestra.run.smithi022.stdout:libcephfs-devel-17.2.6-1165.g93dce5e1.el8.x86_6 70 kB/s | 25 kB 00:00 2023-09-30T00:43:07.122 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:07.123 INFO:teuthology.orchestra.run.smithi022.stdout:Total 69 kB/s | 25 kB 00:00 2023-09-30T00:43:07.123 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:43:07.126 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:43:07.127 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:43:07.128 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:43:07.130 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:43:07.134 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:43:07.188 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:43:07.317 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:07.391 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:07.631 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:07.712 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:08.362 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:08.362 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:08.362 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:43:08.362 INFO:teuthology.orchestra.run.smithi022.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:08.362 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:08.362 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:08.398 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:08.398 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:08.399 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:43:08.399 INFO:teuthology.orchestra.run.smithi006.stdout: libcephfs-devel-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:08.399 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:08.399 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:08.522 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librados2 2023-09-30T00:43:08.526 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librados2 2023-09-30T00:43:09.510 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:04 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:09.514 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:05 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:11.061 INFO:teuthology.orchestra.run.smithi006.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:11.084 INFO:teuthology.orchestra.run.smithi022.stdout:Package librados2-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:11.274 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:11.276 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:11.276 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:11.294 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:11.296 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:11.296 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:11.375 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install librbd1 2023-09-30T00:43:11.410 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install librbd1 2023-09-30T00:43:12.363 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:08 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:12.403 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:07 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:13.912 INFO:teuthology.orchestra.run.smithi006.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:13.994 INFO:teuthology.orchestra.run.smithi022.stdout:Package librbd1-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:14.127 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:14.129 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:14.129 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:14.204 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:14.206 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:14.206 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:14.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rados 2023-09-30T00:43:14.442 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rados 2023-09-30T00:43:15.230 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:11 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:15.427 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:10 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:16.786 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:16.968 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rados-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:16.993 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:16.994 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:16.994 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:17.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rgw 2023-09-30T00:43:17.176 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:17.178 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:17.178 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:17.278 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rgw 2023-09-30T00:43:18.121 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:14 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:18.262 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:13 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:19.708 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:19.801 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rgw-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:19.916 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:19.917 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:20.010 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:20.011 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:20.011 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:20.022 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-cephfs 2023-09-30T00:43:20.125 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-cephfs 2023-09-30T00:43:21.012 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:17 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:21.119 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:16 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:22.566 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:22.671 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-cephfs-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:22.775 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:22.777 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:22.777 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:22.882 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:22.883 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:22.883 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:22.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install python3-rbd 2023-09-30T00:43:22.994 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install python3-rbd 2023-09-30T00:43:23.918 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:19 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:23.984 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:18 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:25.521 INFO:teuthology.orchestra.run.smithi006.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:25.543 INFO:teuthology.orchestra.run.smithi022.stdout:Package python3-rbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 is already installed. 2023-09-30T00:43:25.731 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:25.732 INFO:teuthology.orchestra.run.smithi006.stdout:Nothing to do. 2023-09-30T00:43:25.732 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:25.752 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:25.754 INFO:teuthology.orchestra.run.smithi022.stdout:Nothing to do. 2023-09-30T00:43:25.754 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:25.842 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-fuse 2023-09-30T00:43:25.866 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-fuse 2023-09-30T00:43:26.827 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:22 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:26.842 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:21 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:28.584 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:28.584 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:28.584 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:43:28.585 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:28.586 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 90 k 2023-09-30T00:43:28.586 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 264 k 2023-09-30T00:43:28.586 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:43:28.598 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 90 k 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:28.599 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:43:28.600 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:28.600 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:43:28.600 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:28.600 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 90 k 2023-09-30T00:43:28.600 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 264 k 2023-09-30T00:43:28.601 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:43:28.754 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 536 kB/s | 90 kB 00:00 2023-09-30T00:43:28.755 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:28.755 INFO:teuthology.orchestra.run.smithi006.stdout:Total 531 kB/s | 90 kB 00:00 2023-09-30T00:43:28.756 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:43:28.768 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:43:28.768 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:43:28.827 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:43:28.831 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:43:28.886 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-fuse-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 316 kB/s | 90 kB 00:00 2023-09-30T00:43:28.886 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:28.887 INFO:teuthology.orchestra.run.smithi022.stdout:Total 314 kB/s | 90 kB 00:00 2023-09-30T00:43:28.887 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:43:28.900 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:43:28.900 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:43:28.943 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:43:28.962 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:43:28.965 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:43:29.078 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:43:29.156 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:29.271 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:29.487 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:29.582 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:30.144 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:30.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:30.145 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:43:30.145 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:30.145 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:30.145 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:30.254 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:30.255 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:30.255 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:43:30.255 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-fuse-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:30.255 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:30.255 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:30.378 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-mirror 2023-09-30T00:43:30.464 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-mirror 2023-09-30T00:43:31.367 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:27 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:31.452 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:26 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:33.150 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:33.150 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:33.150 INFO:teuthology.orchestra.run.smithi006.stdout: Package Arch Version Repo Size 2023-09-30T00:43:33.150 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:43:33.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:33.152 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 3.5 M 2023-09-30T00:43:33.152 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 16 M 2023-09-30T00:43:33.152 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:43:33.212 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout: Package Arch Version Repo Size 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 3.5 M 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:43:33.213 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:33.214 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 3.5 M 2023-09-30T00:43:33.214 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 16 M 2023-09-30T00:43:33.214 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:43:33.549 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 8.7 MB/s | 3.5 MB 00:00 2023-09-30T00:43:33.550 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:33.550 INFO:teuthology.orchestra.run.smithi006.stdout:Total 8.7 MB/s | 3.5 MB 00:00 2023-09-30T00:43:33.550 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:43:33.565 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:43:33.565 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:43:33.637 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:43:33.640 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:43:33.671 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-mirror-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 7.6 MB/s | 3.5 MB 00:00 2023-09-30T00:43:33.672 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:33.673 INFO:teuthology.orchestra.run.smithi022.stdout:Total 7.5 MB/s | 3.5 MB 00:00 2023-09-30T00:43:33.673 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:43:33.687 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:43:33.688 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:43:33.761 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:43:33.765 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:43:34.126 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:43:34.245 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:43:34.325 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:34.604 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:34.852 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:35.072 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:35.518 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:35.519 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:35.519 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:43:35.519 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:35.519 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:35.519 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:35.742 DEBUG:teuthology.orchestra.run.smithi006:> sudo yum -y install rbd-nbd 2023-09-30T00:43:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:35.797 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:43:35.797 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-mirror-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:35.798 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:35.798 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:36.045 DEBUG:teuthology.orchestra.run.smithi022:> sudo yum -y install rbd-nbd 2023-09-30T00:43:36.737 INFO:teuthology.orchestra.run.smithi006.stdout:Last metadata expiration check: 0:03:32 ago on Sat 30 Sep 2023 12:40:04 AM UTC. 2023-09-30T00:43:37.035 INFO:teuthology.orchestra.run.smithi022.stdout:Last metadata expiration check: 0:03:32 ago on Sat 30 Sep 2023 12:40:05 AM UTC. 2023-09-30T00:43:38.509 INFO:teuthology.orchestra.run.smithi006.stdout:Dependencies resolved. 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout: Package Architecture Version Repository Size 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout:Installing: 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:38.510 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction Summary 2023-09-30T00:43:38.511 INFO:teuthology.orchestra.run.smithi006.stdout:================================================================================ 2023-09-30T00:43:38.511 INFO:teuthology.orchestra.run.smithi006.stdout:Install 1 Package 2023-09-30T00:43:38.511 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:38.511 INFO:teuthology.orchestra.run.smithi006.stdout:Total download size: 173 k 2023-09-30T00:43:38.511 INFO:teuthology.orchestra.run.smithi006.stdout:Installed size: 552 k 2023-09-30T00:43:38.511 INFO:teuthology.orchestra.run.smithi006.stdout:Downloading Packages: 2023-09-30T00:43:38.710 INFO:teuthology.orchestra.run.smithi006.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 872 kB/s | 173 kB 00:00 2023-09-30T00:43:38.711 INFO:teuthology.orchestra.run.smithi006.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:38.711 INFO:teuthology.orchestra.run.smithi006.stdout:Total 865 kB/s | 173 kB 00:00 2023-09-30T00:43:38.711 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction check 2023-09-30T00:43:38.725 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction check succeeded. 2023-09-30T00:43:38.725 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction test 2023-09-30T00:43:38.784 INFO:teuthology.orchestra.run.smithi006.stdout:Transaction test succeeded. 2023-09-30T00:43:38.787 INFO:teuthology.orchestra.run.smithi006.stdout:Running transaction 2023-09-30T00:43:38.797 INFO:teuthology.orchestra.run.smithi022.stdout:Dependencies resolved. 2023-09-30T00:43:38.797 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:38.797 INFO:teuthology.orchestra.run.smithi022.stdout: Package Architecture Version Repository Size 2023-09-30T00:43:38.797 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout:Installing: 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd x86_64 2:17.2.6-1165.g93dce5e1.el8 ceph 173 k 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction Summary 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout:================================================================================ 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout:Install 1 Package 2023-09-30T00:43:38.798 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:38.799 INFO:teuthology.orchestra.run.smithi022.stdout:Total download size: 173 k 2023-09-30T00:43:38.799 INFO:teuthology.orchestra.run.smithi022.stdout:Installed size: 552 k 2023-09-30T00:43:38.799 INFO:teuthology.orchestra.run.smithi022.stdout:Downloading Packages: 2023-09-30T00:43:38.913 INFO:teuthology.orchestra.run.smithi006.stdout: Preparing : 1/1 2023-09-30T00:43:39.088 INFO:teuthology.orchestra.run.smithi006.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:39.091 INFO:teuthology.orchestra.run.smithi022.stdout:rbd-nbd-17.2.6-1165.g93dce5e1.el8.x86_64.rpm 593 kB/s | 173 kB 00:00 2023-09-30T00:43:39.091 INFO:teuthology.orchestra.run.smithi022.stdout:-------------------------------------------------------------------------------- 2023-09-30T00:43:39.092 INFO:teuthology.orchestra.run.smithi022.stdout:Total 590 kB/s | 173 kB 00:00 2023-09-30T00:43:39.092 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction check 2023-09-30T00:43:39.105 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction check succeeded. 2023-09-30T00:43:39.106 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction test 2023-09-30T00:43:39.165 INFO:teuthology.orchestra.run.smithi022.stdout:Transaction test succeeded. 2023-09-30T00:43:39.169 INFO:teuthology.orchestra.run.smithi022.stdout:Running transaction 2023-09-30T00:43:39.317 INFO:teuthology.orchestra.run.smithi022.stdout: Preparing : 1/1 2023-09-30T00:43:39.502 INFO:teuthology.orchestra.run.smithi022.stdout: Installing : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:39.518 INFO:teuthology.orchestra.run.smithi006.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:39.837 INFO:teuthology.orchestra.run.smithi022.stdout: Running scriptlet: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:40.194 INFO:teuthology.orchestra.run.smithi006.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:40.194 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:40.194 INFO:teuthology.orchestra.run.smithi006.stdout:Installed: 2023-09-30T00:43:40.195 INFO:teuthology.orchestra.run.smithi006.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:40.195 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:40.195 INFO:teuthology.orchestra.run.smithi006.stdout:Complete! 2023-09-30T00:43:40.447 DEBUG:teuthology.parallel:result is None 2023-09-30T00:43:40.521 INFO:teuthology.orchestra.run.smithi022.stdout: Verifying : rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 1/1 2023-09-30T00:43:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:40.522 INFO:teuthology.orchestra.run.smithi022.stdout:Installed: 2023-09-30T00:43:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: rbd-nbd-2:17.2.6-1165.g93dce5e1.el8.x86_64 2023-09-30T00:43:40.522 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:40.522 INFO:teuthology.orchestra.run.smithi022.stdout:Complete! 2023-09-30T00:43:40.783 DEBUG:teuthology.parallel:result is None 2023-09-30T00:43:40.784 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:43:40.935 DEBUG:teuthology.orchestra.run.smithi006:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:43:40.964 INFO:teuthology.orchestra.run.smithi006.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:43:40.965 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:43:40.965 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:43:40.965 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=93dce5e17918f8aca65993a7a2dc40233a47b481 2023-09-30T00:43:41.107 DEBUG:teuthology.orchestra.run.smithi022:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-09-30T00:43:41.136 INFO:teuthology.orchestra.run.smithi022.stdout:17.2.6-1165.g93dce5e1.el8 2023-09-30T00:43:41.137 INFO:teuthology.packaging:The installed version of ceph is 17.2.6-1165.g93dce5e1.el8 2023-09-30T00:43:41.137 INFO:teuthology.task.install:The correct ceph version 17.2.6-1165.g93dce5e1 is installed. 2023-09-30T00:43:41.137 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-09-30T00:43:41.137 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:41.137 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:43:41.173 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:41.174 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-09-30T00:43:41.214 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-09-30T00:43:41.214 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:41.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:43:41.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:43:41.324 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:41.324 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/daemon-helper 2023-09-30T00:43:41.359 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-09-30T00:43:41.434 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-09-30T00:43:41.434 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:41.434 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:43:41.470 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:43:41.544 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:41.544 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/adjust-ulimits 2023-09-30T00:43:41.579 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-09-30T00:43:41.654 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-09-30T00:43:41.655 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:41.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:43:41.690 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:43:41.764 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:41.765 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/usr/bin/stdin-killer 2023-09-30T00:43:41.799 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-09-30T00:43:41.875 INFO:teuthology.run_tasks:Running task ceph... 2023-09-30T00:43:41.943 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2023-09-30T00:43:41.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 777 /var/log/ceph 2023-09-30T00:43:41.946 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 777 /var/log/ceph 2023-09-30T00:43:41.983 INFO:tasks.ceph:Disabling ceph logrotate... 2023-09-30T00:43:41.983 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:43:42.022 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/logrotate.d/ceph 2023-09-30T00:43:42.058 INFO:tasks.ceph:Creating extra log directories... 2023-09-30T00:43:42.059 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:43:42.099 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2023-09-30T00:43:42.145 INFO:tasks.ceph:Creating ceph cluster ceph... 2023-09-30T00:43:42.145 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'slow request', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2023-09-30T00:43:42.145 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719', 'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete}', 'email': 'yweinste@redhat.com', 'first_in_suite': False, 'job_id': '7406719', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.stream', 'overrides': {'admin_socket': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', 'slow request', 'overall HEALTH_', '\\(POOL_APP_NOT_ENABLED\\)'], 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 9810, 'sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'sleep_before_teardown': 0, 'subset': '111/120000', 'suite': 'rados', 'suite_branch': 'wip-yuri4-testing-2023-09-29-0928-quincy', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '93dce5e17918f8aca65993a7a2dc40233a47b481', 'targets': {'smithi006.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnRvJR0L2T9BYkKpGQNJp0fSnvm/Iup0o098MrH+5RqggRpFysYNoYtSqH26RVshsFZM5mKCGbBP/pXIRjBSlo=', 'smithi022.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIS9surE7teWMyjraGAsw5v/FL+wIMOQobe4oi+WNkyXfGPxNue/tZudQy9axQWyCS951ZOY870XlEUUAonQhwo='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1, 'thrash_many': True}}, {'exec': {'client.0': ['ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel', 'ceph_test_rados_delete_pools_parallel']}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '54e62bcbac4e53d9685e08328b790d3b20d71cae', 'timestamp': '2023-09-29_19:38:56', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2830'} 2023-09-30T00:43:42.146 DEBUG:teuthology.orchestra.run.smithi006:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:43:42.173 DEBUG:teuthology.orchestra.run.smithi022:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2023-09-30T00:43:42.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:43:42.236 DEBUG:teuthology.orchestra.run.smithi022:> sudo install -d -m0777 -- /var/run/ceph 2023-09-30T00:43:42.274 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:42.274 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:43:42.332 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:43:42.332 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_1 2023-09-30T00:43:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:43:42.394 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:42.394 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 394050 Links: 1 2023-09-30T00:43:42.394 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:42.395 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:42.395 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-09-30 00:43:34.571708048 +0000 2023-09-30T00:43:42.395 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-09-30 00:38:50.584899758 +0000 2023-09-30T00:43:42.395 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-09-30 00:38:50.584899758 +0000 2023-09-30T00:43:42.395 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-09-30T00:43:42.395 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:43:42.468 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-09-30T00:43:42.468 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-09-30T00:43:42.468 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000227559 s, 2.2 MB/s 2023-09-30T00:43:42.469 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:43:42.534 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_2 2023-09-30T00:43:42.597 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:43:42.597 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 394553 Links: 1 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-09-30 00:43:34.571708048 +0000 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-09-30 00:38:51.147894008 +0000 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-09-30 00:38:51.147894008 +0000 2023-09-30T00:43:42.598 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-09-30T00:43:42.598 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:43:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-09-30T00:43:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-09-30T00:43:42.671 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000238469 s, 2.1 MB/s 2023-09-30T00:43:42.672 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:43:42.737 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_3 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 396420 Links: 1 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-09-30 00:43:34.571708048 +0000 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-09-30 00:38:51.713888228 +0000 2023-09-30T00:43:42.800 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-09-30 00:38:51.713888228 +0000 2023-09-30T00:43:42.801 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-09-30T00:43:42.801 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:43:42.872 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-09-30T00:43:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-09-30T00:43:42.873 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000212521 s, 2.4 MB/s 2023-09-30T00:43:42.874 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:43:42.938 DEBUG:teuthology.orchestra.run.smithi006:> stat /dev/vg_nvme/lv_4 2023-09-30T00:43:43.002 INFO:teuthology.orchestra.run.smithi006.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:43:43.002 INFO:teuthology.orchestra.run.smithi006.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:43.002 INFO:teuthology.orchestra.run.smithi006.stdout:Device: 6h/6d Inode: 383839 Links: 1 2023-09-30T00:43:43.002 INFO:teuthology.orchestra.run.smithi006.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:43.003 INFO:teuthology.orchestra.run.smithi006.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:43.003 INFO:teuthology.orchestra.run.smithi006.stdout:Access: 2023-09-30 00:43:34.571708048 +0000 2023-09-30T00:43:43.003 INFO:teuthology.orchestra.run.smithi006.stdout:Modify: 2023-09-30 00:38:52.275882488 +0000 2023-09-30T00:43:43.003 INFO:teuthology.orchestra.run.smithi006.stdout:Change: 2023-09-30 00:38:52.275882488 +0000 2023-09-30T00:43:43.003 INFO:teuthology.orchestra.run.smithi006.stdout: Birth: - 2023-09-30T00:43:43.003 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:43:43.075 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records in 2023-09-30T00:43:43.075 INFO:teuthology.orchestra.run.smithi006.stderr:1+0 records out 2023-09-30T00:43:43.075 INFO:teuthology.orchestra.run.smithi006.stderr:512 bytes copied, 0.000224403 s, 2.3 MB/s 2023-09-30T00:43:43.076 DEBUG:teuthology.orchestra.run.smithi006:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:43:43.141 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:43.141 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:43.141 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/scratch_devs of=/dev/stdout 2023-09-30T00:43:43.164 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-09-30T00:43:43.164 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_1 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 415304 Links: 1 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-09-30 00:43:34.789561794 +0000 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-09-30 00:38:50.570929273 +0000 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-09-30 00:38:50.570929273 +0000 2023-09-30T00:43:43.228 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-09-30T00:43:43.229 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-09-30T00:43:43.301 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-09-30T00:43:43.301 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-09-30T00:43:43.301 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000132817 s, 3.9 MB/s 2023-09-30T00:43:43.302 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-09-30T00:43:43.369 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_2 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 417286 Links: 1 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-09-30 00:43:34.789561794 +0000 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-09-30 00:38:51.135922209 +0000 2023-09-30T00:43:43.433 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-09-30 00:38:51.135922209 +0000 2023-09-30T00:43:43.434 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-09-30T00:43:43.434 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-09-30T00:43:43.506 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-09-30T00:43:43.507 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-09-30T00:43:43.507 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.00012405 s, 4.1 MB/s 2023-09-30T00:43:43.508 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-09-30T00:43:43.573 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_3 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 415468 Links: 1 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-09-30 00:43:34.789561794 +0000 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-09-30 00:38:51.704915094 +0000 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-09-30 00:38:51.704915094 +0000 2023-09-30T00:43:43.637 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-09-30T00:43:43.638 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-09-30T00:43:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-09-30T00:43:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-09-30T00:43:43.711 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.000130869 s, 3.9 MB/s 2023-09-30T00:43:43.713 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-09-30T00:43:43.778 DEBUG:teuthology.orchestra.run.smithi022:> stat /dev/vg_nvme/lv_4 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout:Device: 6h/6d Inode: 424256 Links: 1 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout:Context: system_u:object_r:device_t:s0 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout:Access: 2023-09-30 00:43:34.789561794 +0000 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout:Modify: 2023-09-30 00:38:52.258908167 +0000 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout:Change: 2023-09-30 00:38:52.258908167 +0000 2023-09-30T00:43:43.841 INFO:teuthology.orchestra.run.smithi022.stdout: Birth: - 2023-09-30T00:43:43.842 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-09-30T00:43:43.915 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records in 2023-09-30T00:43:43.915 INFO:teuthology.orchestra.run.smithi022.stderr:1+0 records out 2023-09-30T00:43:43.915 INFO:teuthology.orchestra.run.smithi022.stderr:512 bytes copied, 0.00013191 s, 3.9 MB/s 2023-09-30T00:43:43.916 DEBUG:teuthology.orchestra.run.smithi022:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-09-30T00:43:43.981 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:43.981 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T00:43:43.981 INFO:tasks.ceph:Generating config... 2023-09-30T00:43:43.982 INFO:tasks.ceph:[client] debug monc = 20 2023-09-30T00:43:43.982 INFO:tasks.ceph:[client] debug ms = 1 2023-09-30T00:43:43.982 INFO:tasks.ceph:[global] mon client directed command retry = 5 2023-09-30T00:43:43.982 INFO:tasks.ceph:[global] mon election default strategy = 1 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms bind msgr1 = False 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms bind msgr2 = True 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms inject delay max = 1 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms inject delay probability = 0.005 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms inject delay type = mon 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms inject internal delays = 0.002 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms inject socket failures = 2500 2023-09-30T00:43:43.983 INFO:tasks.ceph:[global] ms type = async 2023-09-30T00:43:43.983 INFO:tasks.ceph:[mgr] debug mgr = 20 2023-09-30T00:43:43.983 INFO:tasks.ceph:[mgr] debug monc = 10 2023-09-30T00:43:43.983 INFO:tasks.ceph:[mgr] debug ms = 1 2023-09-30T00:43:43.983 INFO:tasks.ceph:[mon] debug mon = 20 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] debug ms = 1 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] debug paxos = 20 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2023-09-30T00:43:43.984 INFO:tasks.ceph:[mon] mon scrub interval = 300 2023-09-30T00:43:43.985 INFO:tasks.ceph:[mon] paxos min = 10 2023-09-30T00:43:43.985 INFO:tasks.ceph:[mon] paxos service trim min = 5 2023-09-30T00:43:43.985 INFO:tasks.ceph:[mon] paxos trim min = 10 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] debug bluefs = 20 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] debug bluestore = 20 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] debug ms = 1 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] debug osd = 20 2023-09-30T00:43:43.985 INFO:tasks.ceph:[osd] debug rocksdb = 10 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd op queue = debug_random 2023-09-30T00:43:43.986 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2023-09-30T00:43:43.986 INFO:tasks.ceph:Setting up mon.a... 2023-09-30T00:43:43.987 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2023-09-30T00:43:44.043 INFO:teuthology.orchestra.run.smithi006.stdout:creating /etc/ceph/ceph.keyring 2023-09-30T00:43:44.045 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2023-09-30T00:43:44.095 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:43:44.127 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.6'), ('mon.c', '[v2:172.21.15.6:3301,v1:172.21.15.6:6790]'), ('mon.b', '172.21.15.22')] 2023-09-30T00:43:44.127 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.6,[v2:172.21.15.6:3301,v1:172.21.15.6:6790],172.21.15.22', 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr1': False, 'ms bind msgr2': True, 'ms inject delay max': 1, 'ms inject delay probability': 0.005, 'ms inject delay type': 'mon', 'ms inject internal delays': 0.002, 'ms inject socket failures': 2500, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2023-09-30T00:43:44.128 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:44.128 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:43:44.191 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.6 --addv c '[v2:172.21.15.6:3301,v1:172.21.15.6:6790]' --add b 172.21.15.22 --print /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:44.285 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:44.285 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: generated fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:setting min_mon_release = octopus 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:epoch 0 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:min_mon_release 15 (octopus) 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:election_strategy: 1 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:43:44.286 INFO:teuthology.orchestra.run.smithi006.stdout:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:43:44.287 INFO:teuthology.orchestra.run.smithi006.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2023-09-30T00:43:44.288 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2023-09-30T00:43:44.309 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID cb933378-6423-4c15-869d-4727cefb664a... 2023-09-30T00:43:44.310 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:43:44.351 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2023-09-30T00:43:44.406 INFO:teuthology.orchestra.run.smithi006.stdout:[global] 2023-09-30T00:43:44.406 INFO:teuthology.orchestra.run.smithi006.stdout: chdir = "" 2023-09-30T00:43:44.406 INFO:teuthology.orchestra.run.smithi006.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:43:44.406 INFO:teuthology.orchestra.run.smithi006.stdout: auth supported = cephx 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: filestore xattr use omap = true 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: mon clock drift allowed = 1.000 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: osd crush chooseleaf type = 0 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: auth debug = true 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on old message = true 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: ms die on bug = true 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.407 INFO:teuthology.orchestra.run.smithi006.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: mon pg warn max object skew = 0 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default size = 2 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow primary affinity = true 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd allow pg remap = true 2023-09-30T00:43:44.408 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on no sortbitwise = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon warn on too few osds = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: mon_allow_pool_size_one = true 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.409 INFO:teuthology.orchestra.run.smithi006.stdout: osd default data pool replay window = 5 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: mon allow pool delete = true 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: mon cluster log file level = debug 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: debug asserts on shutdown = true 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: mon health detail to clog = false 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: mon host = "172.21.15.6,[v2:172.21.15.6:3301,v1:172.21.15.6:6790],172.21.15.22" 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: mon client directed command retry = 5 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: mon election default strategy = 1 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: ms bind msgr1 = False 2023-09-30T00:43:44.410 INFO:teuthology.orchestra.run.smithi006.stdout: ms bind msgr2 = True 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay max = 1 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay probability = 0.005 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject delay type = mon 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject internal delays = 0.002 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: ms inject socket failures = 2500 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: ms type = async 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: fsid = cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout:[osd] 2023-09-30T00:43:44.411 INFO:teuthology.orchestra.run.smithi006.stdout: osd journal size = 100 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub load threshold = 5.0 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd scrub max interval = 600 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd recover clone overlap = true 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd recovery max chunk = 1048576 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug shutdown = true 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug op order = true 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify stray on activate = true 2023-09-30T00:43:44.412 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: osd open classes on start = true 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug pg log writeout = true 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: osd map max advance = 10 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: journal zero on create = true 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: filestore apply finisher threads = 3 2023-09-30T00:43:44.413 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: bdev debug aio = true 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug misdirected ops = true 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore block size = 96636764160 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression algorithm = snappy 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore compression mode = aggressive 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore fsck on mount = True 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: bluestore zero block detection = True 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluefs = 20 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: debug bluestore = 20 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: debug osd = 20 2023-09-30T00:43:44.414 INFO:teuthology.orchestra.run.smithi006.stdout: debug rocksdb = 10 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd full ratio = 0.9 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify cached snaps = True 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd debug verify missing on start = True 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd mclock override recovery settings = True 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd objectstore = bluestore 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue = debug_random 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: osd op queue cut off = debug_random 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.415 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr] 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug mgr = 20 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: mgr/telemetry/nag = false 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 10 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout:[mon] 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug mon = 20 2023-09-30T00:43:44.416 INFO:teuthology.orchestra.run.smithi006.stdout: debug paxos = 20 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: debug auth = 20 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: mon data avail warn = 5 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: mon mgr mkfs grace = 240 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd reporter subtree level = osd 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: mon osd prime pg temp = true 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:43:44.417 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: # 1m isn't quite enough 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon min osdmap epochs = 25 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune interval = 2 2023-09-30T00:43:44.418 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune min = 15 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub inject crc mismatch = 0.01 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub inject missing keys = 0.05 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: mon scrub interval = 300 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: paxos min = 10 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: paxos service trim min = 5 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: paxos trim min = 10 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout:[client] 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: rgw cache enabled = true 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable ops log = true 2023-09-30T00:43:44.419 INFO:teuthology.orchestra.run.smithi006.stdout: rgw enable usage log = true 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout: debug monc = 20 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout: debug ms = 1 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.a] 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.c] 2023-09-30T00:43:44.420 INFO:teuthology.orchestra.run.smithi006.stdout:[mon.b] 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout:[global] 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: chdir = "" 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: auth supported = cephx 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: filestore xattr use omap = true 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: mon clock drift allowed = 1.000 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.421 INFO:teuthology.orchestra.run.smithi022.stdout: osd crush chooseleaf type = 0 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: auth debug = true 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on old message = true 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: ms die on bug = true 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: mon max pg per osd = 10000 # >= luminous 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: mon pg warn max object skew = 0 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: # disable pg_autoscaler by default for new pools 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: osd_pool_default_pg_autoscale_mode = off 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.422 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default size = 2 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow primary affinity = true 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd allow pg remap = true 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on legacy crush tunables = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on crush straw calc version zero = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on no sortbitwise = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on osd down out interval zero = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon warn on too few osds = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_pool_no_redundancy = false 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: mon_allow_pool_size_one = true 2023-09-30T00:43:44.423 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: osd default data pool replay window = 5 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: mon allow pool delete = true 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: mon cluster log file level = debug 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: debug asserts on shutdown = true 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: mon health detail to clog = false 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: mon host = "172.21.15.6,[v2:172.21.15.6:3301,v1:172.21.15.6:6790],172.21.15.22" 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: mon client directed command retry = 5 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: mon election default strategy = 1 2023-09-30T00:43:44.424 INFO:teuthology.orchestra.run.smithi022.stdout: ms bind msgr1 = False 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms bind msgr2 = True 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject delay max = 1 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject delay probability = 0.005 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject delay type = mon 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject internal delays = 0.002 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms inject socket failures = 2500 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: ms type = async 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: fsid = cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout:[osd] 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: osd journal size = 100 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.425 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub load threshold = 5.0 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd scrub max interval = 600 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock profile = high_recovery_ops 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd recover clone overlap = true 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd recovery max chunk = 1048576 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug shutdown = true 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug op order = true 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify stray on activate = true 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd open classes on start = true 2023-09-30T00:43:44.426 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug pg log writeout = true 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: osd deep scrub update digest min age = 30 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: osd map max advance = 10 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: journal zero on create = true 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: filestore ondisk finisher threads = 3 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: filestore apply finisher threads = 3 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: bdev debug aio = true 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug misdirected ops = true 2023-09-30T00:43:44.427 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore block size = 96636764160 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore compression algorithm = snappy 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore compression mode = aggressive 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore fsck on mount = True 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: bluestore zero block detection = True 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluefs = 20 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: debug bluestore = 20 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: debug osd = 20 2023-09-30T00:43:44.428 INFO:teuthology.orchestra.run.smithi022.stdout: debug rocksdb = 10 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd backfillfull_ratio = 0.85 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd full ratio = 0.9 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd nearfull ratio = 0.8 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify cached snaps = True 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd debug verify missing on start = True 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd failsafe full ratio = 0.95 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd mclock override recovery settings = True 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd objectstore = bluestore 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue = debug_random 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: osd op queue cut off = debug_random 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.429 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr] 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug mgr = 20 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: mgr/telemetry/nag = false 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 10 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout:[mon] 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-09-30T00:43:44.430 INFO:teuthology.orchestra.run.smithi022.stdout: debug mon = 20 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: debug paxos = 20 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: debug auth = 20 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: mon data avail warn = 5 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: mon mgr mkfs grace = 240 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min pgs per osd = 4 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd reporter subtree level = osd 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: mon osd prime pg temp = true 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: mon reweight min bytes per osd = 10 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: # rotate auth tickets quickly to exercise renewal paths 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: auth mon ticket ttl = 660 # 11m 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: auth service ticket ttl = 240 # 4m 2023-09-30T00:43:44.431 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: # don't complain about insecure global_id in the test suite 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim = false 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: # 1m isn't quite enough 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon_down_mkfs_grace = 2m 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon_warn_on_filestore_osds = false 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon min osdmap epochs = 25 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune interval = 2 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune min = 15 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon osdmap full prune txsize = 2 2023-09-30T00:43:44.432 INFO:teuthology.orchestra.run.smithi022.stdout: mon scrub inject crc mismatch = 0.01 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: mon scrub inject missing keys = 0.05 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: mon scrub interval = 300 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: paxos min = 10 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: paxos service trim min = 5 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: paxos trim min = 10 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout:[client] 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: rgw cache enabled = true 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable ops log = true 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: rgw enable usage log = true 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2023-09-30T00:43:44.433 INFO:teuthology.orchestra.run.smithi022.stdout: debug monc = 20 2023-09-30T00:43:44.434 INFO:teuthology.orchestra.run.smithi022.stdout: debug ms = 1 2023-09-30T00:43:44.434 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.a] 2023-09-30T00:43:44.434 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.c] 2023-09-30T00:43:44.434 INFO:teuthology.orchestra.run.smithi022.stdout:[mon.b] 2023-09-30T00:43:44.435 INFO:tasks.ceph:Creating admin key on mon.a... 2023-09-30T00:43:44.435 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2023-09-30T00:43:44.539 INFO:tasks.ceph:Copying monmap to all nodes... 2023-09-30T00:43:44.540 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:44.540 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2023-09-30T00:43:44.561 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:44.561 DEBUG:teuthology.orchestra.run.smithi006:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2023-09-30T00:43:44.624 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:43:44.624 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:44.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:43:44.624 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:43:44.711 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:43:44.711 DEBUG:teuthology.orchestra.run.smithi006:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:44.774 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:43:44.774 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:44.774 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd of=/etc/ceph/ceph.keyring 2023-09-30T00:43:44.774 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0644 /etc/ceph/ceph.keyring 2023-09-30T00:43:44.821 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:43:44.821 DEBUG:teuthology.orchestra.run.smithi022:> dd of=/home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:44.884 INFO:tasks.ceph:Setting up mon nodes... 2023-09-30T00:43:44.884 INFO:tasks.ceph:Setting up mgr nodes... 2023-09-30T00:43:44.884 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:43:44.994 INFO:teuthology.orchestra.run.smithi022.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2023-09-30T00:43:44.997 INFO:tasks.ceph:Setting up mds nodes... 2023-09-30T00:43:44.997 INFO:tasks.ceph_client:Setting up client nodes... 2023-09-30T00:43:44.998 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-09-30T00:43:45.046 INFO:teuthology.orchestra.run.smithi022.stdout:creating /etc/ceph/ceph.client.0.keyring 2023-09-30T00:43:45.060 INFO:tasks.ceph:Running mkfs on osd nodes... 2023-09-30T00:43:45.061 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi006.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi022.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2023-09-30T00:43:45.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2023-09-30T00:43:45.096 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:45.096 INFO:tasks.ceph:role: osd.0 2023-09-30T00:43:45.096 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:43:45.097 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:43:45.178 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:43:45.179 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:43:45.610 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-09-30T00:43:45.610 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-09-30T00:43:45.611 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2023-09-30T00:43:45.767 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2023-09-30T00:43:45.809 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2023-09-30T00:43:45.885 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:45.885 INFO:tasks.ceph:role: osd.1 2023-09-30T00:43:45.886 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:43:45.886 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:43:45.968 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:43:45.968 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:43:45.968 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:43:45.968 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:43:45.968 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:43:45.969 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:43:45.969 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:43:45.969 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:43:45.969 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:43:45.969 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:43:46.385 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-09-30T00:43:46.387 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-09-30T00:43:46.387 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2023-09-30T00:43:46.508 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2023-09-30T00:43:46.547 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2023-09-30T00:43:46.622 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:46.623 INFO:tasks.ceph:role: osd.2 2023-09-30T00:43:46.623 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:43:46.623 DEBUG:teuthology.orchestra.run.smithi006:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:43:46.705 INFO:teuthology.orchestra.run.smithi006.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:43:46.706 INFO:teuthology.orchestra.run.smithi006.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:43:47.146 INFO:teuthology.orchestra.run.smithi006.stdout:Discarding blocks...Done. 2023-09-30T00:43:47.148 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi006.front.sepia.ceph.com -o noatime 2023-09-30T00:43:47.148 DEBUG:teuthology.orchestra.run.smithi006:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2023-09-30T00:43:47.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2023-09-30T00:43:47.372 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:47.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:47.481+0000 7ff4fbb26540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2023-09-30T00:43:47.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:47.484+0000 7ff4fbb26540 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2023-09-30T00:43:47.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:47.484+0000 7ff4fbb26540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:43:47.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:47.484+0000 7ff4fbb26540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:43:47.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:47.484+0000 7ff4fbb26540 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2023-09-30T00:43:47.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:47.487+0000 7ff4fbb26540 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2023-09-30T00:43:48.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:48.015+0000 7ff4fbb26540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:48.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:48.785+0000 7ff4fbb26540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:49.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:49.548+0000 7ff4fbb26540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:50.347 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2023-09-30T00:43:50.382 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:50.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:50.477+0000 7f29a4b62540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2023-09-30T00:43:50.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:50.477+0000 7f29a4b62540 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2023-09-30T00:43:50.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:50.477+0000 7f29a4b62540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:43:50.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:50.477+0000 7f29a4b62540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:43:50.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:50.477+0000 7f29a4b62540 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2023-09-30T00:43:50.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:50.478+0000 7f29a4b62540 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2023-09-30T00:43:51.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:51.002+0000 7f29a4b62540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:51.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:51.766+0000 7f29a4b62540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:52.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:52.532+0000 7f29a4b62540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:53.327 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2023-09-30T00:43:53.363 DEBUG:teuthology.orchestra.run.smithi006:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:53.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.456+0000 7f4653dd9540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2023-09-30T00:43:53.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.456+0000 7f4653dd9540 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2023-09-30T00:43:53.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.456+0000 7f4653dd9540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:43:53.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.456+0000 7f4653dd9540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:43:53.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.456+0000 7f4653dd9540 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2023-09-30T00:43:53.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.457+0000 7f4653dd9540 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2023-09-30T00:43:53.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:53.985+0000 7f4653dd9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:54.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:54.748+0000 7f4653dd9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:55.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:43:55.511+0000 7f4653dd9540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:56.306 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2023-09-30T00:43:56.342 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2023-09-30T00:43:56.378 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:56.378 INFO:tasks.ceph:role: osd.3 2023-09-30T00:43:56.379 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:43:56.379 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2023-09-30T00:43:56.454 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:43:56.454 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:43:56.454 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:43:56.454 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:43:56.455 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:43:56.455 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:43:56.455 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:43:56.455 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:43:56.455 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:43:56.455 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:43:56.477 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-09-30T00:43:56.479 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-09-30T00:43:56.479 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2023-09-30T00:43:56.571 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2023-09-30T00:43:56.609 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2023-09-30T00:43:56.685 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:56.685 INFO:tasks.ceph:role: osd.4 2023-09-30T00:43:56.685 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:43:56.685 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2023-09-30T00:43:56.760 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:43:56.761 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:43:56.783 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-09-30T00:43:56.786 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-09-30T00:43:56.786 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2023-09-30T00:43:56.875 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2023-09-30T00:43:56.912 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2023-09-30T00:43:56.987 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2023-09-30T00:43:56.987 INFO:tasks.ceph:role: osd.5 2023-09-30T00:43:56.987 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:43:56.987 DEBUG:teuthology.orchestra.run.smithi022:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2023-09-30T00:43:57.062 INFO:teuthology.orchestra.run.smithi022.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2023-09-30T00:43:57.062 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 attr=2, projid32bit=1 2023-09-30T00:43:57.062 INFO:teuthology.orchestra.run.smithi022.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2023-09-30T00:43:57.062 INFO:teuthology.orchestra.run.smithi022.stdout: = reflink=1 bigtime=0 inobtcount=0 2023-09-30T00:43:57.063 INFO:teuthology.orchestra.run.smithi022.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2023-09-30T00:43:57.063 INFO:teuthology.orchestra.run.smithi022.stdout: = sunit=0 swidth=0 blks 2023-09-30T00:43:57.063 INFO:teuthology.orchestra.run.smithi022.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2023-09-30T00:43:57.063 INFO:teuthology.orchestra.run.smithi022.stdout:log =internal log bsize=4096 blocks=11443, version=2 2023-09-30T00:43:57.063 INFO:teuthology.orchestra.run.smithi022.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2023-09-30T00:43:57.063 INFO:teuthology.orchestra.run.smithi022.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2023-09-30T00:43:57.084 INFO:teuthology.orchestra.run.smithi022.stdout:Discarding blocks...Done. 2023-09-30T00:43:57.086 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi022.front.sepia.ceph.com -o noatime 2023-09-30T00:43:57.086 DEBUG:teuthology.orchestra.run.smithi022:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2023-09-30T00:43:57.174 DEBUG:teuthology.orchestra.run.smithi022:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2023-09-30T00:43:57.212 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:43:57.316 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.316+0000 7fa3d522c540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2023-09-30T00:43:57.317 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.316+0000 7fa3d522c540 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2023-09-30T00:43:57.317 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.316+0000 7fa3d522c540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:43:57.317 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.316+0000 7fa3d522c540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:43:57.317 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.316+0000 7fa3d522c540 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2023-09-30T00:43:57.317 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.316+0000 7fa3d522c540 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2023-09-30T00:43:57.841 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:57.840+0000 7fa3d522c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:58.603 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:58.602+0000 7fa3d522c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:43:59.365 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:43:59.364+0000 7fa3d522c540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:00.162 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2023-09-30T00:44:00.197 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:44:00.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.291+0000 7f447375f540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2023-09-30T00:44:00.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.291+0000 7f447375f540 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2023-09-30T00:44:00.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.291+0000 7f447375f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:44:00.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.291+0000 7f447375f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:44:00.292 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.291+0000 7f447375f540 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2023-09-30T00:44:00.293 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.292+0000 7f447375f540 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2023-09-30T00:44:00.817 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:00.816+0000 7f447375f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:01.578 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:01.578+0000 7f447375f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:02.340 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:02.340+0000 7f447375f540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:03.137 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2023-09-30T00:44:03.172 DEBUG:teuthology.orchestra.run.smithi022:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:44:03.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.266+0000 7f88c9d1a540 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2023-09-30T00:44:03.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.267+0000 7f88c9d1a540 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2023-09-30T00:44:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.267+0000 7f88c9d1a540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:44:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.267+0000 7f88c9d1a540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:44:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.267+0000 7f88c9d1a540 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2023-09-30T00:44:03.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.267+0000 7f88c9d1a540 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2023-09-30T00:44:03.792 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:03.792+0000 7f88c9d1a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:04.553 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:04.553+0000 7f88c9d1a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:05.316 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:05.315+0000 7f88c9d1a540 -1 bluestore::NCB::__restore_allocator::Failed open_for_read with error-code -2 2023-09-30T00:44:06.109 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2023-09-30T00:44:06.145 INFO:tasks.ceph:Reading keys from all nodes... 2023-09-30T00:44:06.145 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:44:06.145 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2023-09-30T00:44:06.180 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:44:06.181 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2023-09-30T00:44:06.255 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:44:06.255 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2023-09-30T00:44:06.330 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:06.330 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2023-09-30T00:44:06.365 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:06.365 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2023-09-30T00:44:06.440 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:06.441 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2023-09-30T00:44:06.517 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:06.517 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2023-09-30T00:44:06.592 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:06.592 DEBUG:teuthology.orchestra.run.smithi022:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2023-09-30T00:44:06.655 INFO:tasks.ceph:Adding keys to all mons... 2023-09-30T00:44:06.655 DEBUG:teuthology.orchestra.run.smithi006:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:44:06.658 DEBUG:teuthology.orchestra.run.smithi022:> sudo tee -a /etc/ceph/ceph.keyring 2023-09-30T00:44:06.698 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.0] 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBDbxdlkl/IHBAACgiKJqaRQ6ENe/TUCPK46Q== 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.1] 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBGbxdlY1eJHBAAPEqlEVQubSBXoMCsNB5RGA== 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.2] 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBJbxdlx4FFGxAAqtpmzePxM9YXnHpJCZUcAA== 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout:[mgr.x] 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBAbxdlm8RQOxAA8KPyeDOnd8hMf96L/PtWow== 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.3] 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBNbxdlLtbiEhAA6M3Da5yifeRBJu4xLvIrNQ== 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.4] 2023-09-30T00:44:06.699 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBQbxdlQ+ltERAAO7fp8iDeGWN9ifLXc3R74A== 2023-09-30T00:44:06.700 INFO:teuthology.orchestra.run.smithi006.stdout:[osd.5] 2023-09-30T00:44:06.700 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBTbxdloqv1DxAARJML4qBqVbLnF1ysBeiixg== 2023-09-30T00:44:06.700 INFO:teuthology.orchestra.run.smithi006.stdout:[client.0] 2023-09-30T00:44:06.700 INFO:teuthology.orchestra.run.smithi006.stdout: key = AQBBbxdlw6vEAhAAQ5ypE+0f8UA3L/C23jiCgQ== 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.0] 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBDbxdlkl/IHBAACgiKJqaRQ6ENe/TUCPK46Q== 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.1] 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBGbxdlY1eJHBAAPEqlEVQubSBXoMCsNB5RGA== 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.2] 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBJbxdlx4FFGxAAqtpmzePxM9YXnHpJCZUcAA== 2023-09-30T00:44:06.729 INFO:teuthology.orchestra.run.smithi022.stdout:[mgr.x] 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBAbxdlm8RQOxAA8KPyeDOnd8hMf96L/PtWow== 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.3] 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBNbxdlLtbiEhAA6M3Da5yifeRBJu4xLvIrNQ== 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.4] 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBQbxdlQ+ltERAAO7fp8iDeGWN9ifLXc3R74A== 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout:[osd.5] 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBTbxdloqv1DxAARJML4qBqVbLnF1ysBeiixg== 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout:[client.0] 2023-09-30T00:44:06.730 INFO:teuthology.orchestra.run.smithi022.stdout: key = AQBBbxdlw6vEAhAAQ5ypE+0f8UA3L/C23jiCgQ== 2023-09-30T00:44:06.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:06.743 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:06.833 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:06.835 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:06.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:06.897 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:06.955 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:44:06.957 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2023-09-30T00:44:07.014 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:07.016 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:07.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:07.075 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:07.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:07.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2023-09-30T00:44:07.188 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:44:07.190 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2023-09-30T00:44:07.249 INFO:tasks.ceph:Running mkfs on mon nodes... 2023-09-30T00:44:07.249 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2023-09-30T00:44:07.281 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:44:07.384 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2023-09-30T00:44:07.416 DEBUG:teuthology.orchestra.run.smithi006:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2023-09-30T00:44:07.489 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:44:07.599 DEBUG:teuthology.orchestra.run.smithi006:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2023-09-30T00:44:07.632 DEBUG:teuthology.orchestra.run.smithi022:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2023-09-30T00:44:07.667 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2023-09-30T00:44:07.774 DEBUG:teuthology.orchestra.run.smithi022:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2023-09-30T00:44:07.807 DEBUG:teuthology.orchestra.run.smithi006:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:44:07.809 DEBUG:teuthology.orchestra.run.smithi022:> rm -- /home/ubuntu/cephtest/ceph.monmap 2023-09-30T00:44:07.870 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2023-09-30T00:44:07.871 INFO:tasks.ceph.mon.a:Restarting daemon 2023-09-30T00:44:07.871 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2023-09-30T00:44:07.873 INFO:tasks.ceph.mon.a:Started 2023-09-30T00:44:07.873 INFO:tasks.ceph.mon.c:Restarting daemon 2023-09-30T00:44:07.874 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2023-09-30T00:44:07.875 INFO:tasks.ceph.mon.c:Started 2023-09-30T00:44:07.876 INFO:tasks.ceph.mon.b:Restarting daemon 2023-09-30T00:44:07.876 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2023-09-30T00:44:07.913 INFO:tasks.ceph.mon.b:Started 2023-09-30T00:44:07.913 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2023-09-30T00:44:07.913 INFO:tasks.ceph.mgr.x:Restarting daemon 2023-09-30T00:44:07.913 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2023-09-30T00:44:07.915 INFO:tasks.ceph.mgr.x:Started 2023-09-30T00:44:07.915 INFO:tasks.ceph:Setting crush tunables to default 2023-09-30T00:44:07.916 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd crush tunables default 2023-09-30T00:44:08.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.050+0000 7f1a925d9700 10 monclient: get_monmap_and_config 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.050+0000 7f1a925d9700 10 monclient: build_initial_monmap 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.050+0000 7f1a925d9700 10 monclient: monmap: 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:08.051662+0000 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:08.051662+0000 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:08.052 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 Processor -- start 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 -- start start 2023-09-30T00:44:08.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient: init 2023-09-30T00:44:08.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:08.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-09-30T00:44:08.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:08.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): picked mon.noname-c con 0x7f1a8c098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): picked mon.noname-b con 0x7f1a8c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1a8c098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c0a1000 con 0x7f1a8c098e90 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a0a50 con 0x7f1a8c098670 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a0730 con 0x7f1a8c098a40 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a925d9700 10 monclient(hunting): authenticate will time out at 1244.349850s 2023-09-30T00:44:08.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.051+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:33588/0 (socket says 172.21.15.6:33588) 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 learned_addr learned my addr 172.21.15.6:0/4080131734 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request con 0x7f1a8c098a40 auth_method 0 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:08.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a8c098670 auth_method 0 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.052+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:44:08.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.054+0000 7f1a90b76700 1 -- 172.21.15.6:0/4080131734 >> v1:172.21.15.22:6789/0 conn(0x7f1a8c098e90 legacy=0x7f1a8c12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:08.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request con 0x7f1a8c098a40 auth_method 0 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a8c098670 auth_method 0 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:08.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:08.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:08.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:08.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:08.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.253+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:44:08.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request con 0x7f1a8c098a40 auth_method 0 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:08.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a8c098670 auth_method 0 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:08.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:08.654+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:44:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request con 0x7f1a8c098a40 auth_method 0 2023-09-30T00:44:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:09.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a8c098670 auth_method 0 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:09.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:44:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:09.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:09.455+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:44:11.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient(hunting): tick 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 monclient(hunting): continuing hunt 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient(hunting): _reopen_session rank -1 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c12ede0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a8c12ede0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 msgr2=0x7f1a8c132d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.22:3300/0 conn(0x7f1a8c098670 0x7f1a8c132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 -- 172.21.15.6:0/4080131734 >> v1:172.21.15.22:6789/0 conn(0x7f1a8c098e90 legacy=0x7f1a8c12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:11.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient: _add_conns ranks=[3,0] 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient(hunting): picked mon.noname-a con 0x7f1a78003790 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f1a780062f0 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 -- 172.21.15.6:0/4080131734 --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a8c068df0 con 0x7f1a780062f0 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 1 -- 172.21.15.6:0/4080131734 --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a0a50 con 0x7f1a78003790 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a89ffb700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:11.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a78003790 auth_method 0 2023-09-30T00:44:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:11.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.052+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:44:11.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.054+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/4080131734 >> v1:172.21.15.6:6789/0 conn(0x7f1a780062f0 legacy=0x7f1a78008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:11.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:11.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a78003790 auth_method 0 2023-09-30T00:44:11.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:11.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:11.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:11.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:11.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:11.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:11.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.253+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2023-09-30T00:44:11.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a78003790 auth_method 0 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:11.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:11.654+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2023-09-30T00:44:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request con 0x7f1a78003790 auth_method 0 2023-09-30T00:44:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:12.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:12.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:12.455+0000 7f1a8bfff700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2023-09-30T00:44:13.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.370+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 <== mon.0 v1:172.21.15.6:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3593340598 0 0) 0x7f1a8c068df0 con 0x7f1a780062f0 2023-09-30T00:44:13.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:13.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:13.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): my global_id is 4098 2023-09-30T00:44:13.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 --> v1:172.21.15.6:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a74003570 con 0x7f1a780062f0 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 <== mon.0 v1:172.21.15.6:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959163667 0 0) 0x7f1a74003570 con 0x7f1a780062f0 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 msgr2=0x7f1a78005c00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 --2- 172.21.15.6:0/4080131734 >> v2:172.21.15.6:3300/0 conn(0x7f1a78003790 0x7f1a78005c00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 monclient: found mon.noname-a-legacy 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: _send_mon_message to mon.noname-a-legacy at v1:172.21.15.6:6789/0 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 --> v1:172.21.15.6:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a7800b070 con 0x7f1a780062f0 2023-09-30T00:44:13.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: _finish_auth 0 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 <== mon.0 v1:172.21.15.6:6789/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (unknown 3401003695 0 0) 0x7f1a8000c680 con 0x7f1a780062f0 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: got monmap 1 from mon.noname-a-legacy (according to old e1) 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: dump: 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.378 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 monclient: mon.a has (v2) addrs [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] but i'm connected to v1:172.21.15.6:6789/0, reconnecting 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 >> v1:172.21.15.6:6789/0 conn(0x7f1a780062f0 legacy=0x7f1a78008780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a90b76700 1 -- 172.21.15.6:0/4080131734 reap_dead start 2023-09-30T00:44:13.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a74003ab0 0x7f1a74003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): picked mon.b con 0x7f1a74003ab0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a74004810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): picked mon.c con 0x7f1a8c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a740052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): picked mon.a con 0x7f1a74004e70 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c0a0a50 con 0x7f1a74004e70 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7400bd00 con 0x7f1a74003ab0 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a7400bec0 con 0x7f1a8c098a40 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a8affd700 10 monclient(hunting): _finish_auth 0 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.371+0000 7f1a90b76700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a740052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a74004810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): get_auth_request con 0x7f1a74004e70 auth_method 0 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request con 0x7f1a8c098a40 auth_method 0 2023-09-30T00:44:13.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient(hunting): handle_auth_done global_id 4098 payload 293 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 msgr2=0x7f1a74004810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:13.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a74004810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a74003ab0 msgr2=0x7f1a74003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a74003ab0 0x7f1a74003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient: _finish_hunting 0 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 1 monclient: found mon.a 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 1 -- 172.21.15.6:0/4080131734 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a74003570 con 0x7f1a74004e70 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 10 monclient: _finish_auth 0 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a90b76700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a925d9700 5 monclient: authenticate success, global_id 4098 2023-09-30T00:44:13.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a925d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.372+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a74004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.374+0000 7f1a90b76700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 msgr2=0x7f1a740052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.374+0000 7f1a90b76700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a740052e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0a6590 tx=0x7f1a84007f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=153fc538c65b84ed server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.374+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a84005b90 con 0x7f1a74004e70 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: dump: 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:13.384 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a925d9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a84005cf0 con 0x7f1a74004e70 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:13.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 1 -- 172.21.15.6:0/4080131734 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8400ca30 con 0x7f1a74004e70 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a8affd700 10 monclient: dump: 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:13.386 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a925d9700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.375+0000 7f1a925d9700 4 set_mon_vals no callback set 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 10 monclient: shutdown 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 -- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 msgr2=0x7f1a740052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a740052e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f1a8c0a6590 tx=0x7f1a84007f30 comp rx=0 tx=0).stop 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 -- 172.21.15.6:0/4080131734 shutdown_connections 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a740052e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a74003ab0 0x7f1a74003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 --2- 172.21.15.6:0/4080131734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a8c098a40 0x7f1a74004810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 -- 172.21.15.6:0/4080131734 >> 172.21.15.6:0/4080131734 conn(0x7f1a8c0a5480 msgr2=0x7f1a8c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:13.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 -- 172.21.15.6:0/4080131734 shutdown_connections 2023-09-30T00:44:13.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.376+0000 7f1a925d9700 1 -- 172.21.15.6:0/4080131734 wait complete. 2023-09-30T00:44:13.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.377+0000 7f1a925d9700 10 monclient: build_initial_monmap 2023-09-30T00:44:13.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.377+0000 7f1a925d9700 10 monclient: monmap: 2023-09-30T00:44:13.392 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:13.392 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.377+0000 7f1a925d9700 1 Processor -- start 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 1 -- start start 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 10 monclient: init 2023-09-30T00:44:13.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a74003ab0 0x7f1a8c141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 10 monclient(hunting): picked mon.noname-c con 0x7f1a74003ab0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a8c141990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 10 monclient(hunting): picked mon.noname-a con 0x7f1a74004e70 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.378+0000 7f1a925d9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a8c098a40 0x7f1a8c141ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 10 monclient(hunting): picked mon.noname-b con 0x7f1a8c098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c12f260 con 0x7f1a74004e70 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c1301b0 con 0x7f1a8c098a40 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8c12ee20 con 0x7f1a74003ab0 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a925d9700 10 monclient(hunting): authenticate will time out at 1249.677041s 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a8c141990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a8c141990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35298/0 (socket says 172.21.15.6:35298) 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/430427971 learned_addr learned my addr 172.21.15.6:0/430427971 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request con 0x7f1a74004e70 auth_method 0 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:13.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a8c098a40 0x7f1a8c141ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): get_auth_request con 0x7f1a8c098a40 auth_method 0 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a90b76700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient(hunting): handle_auth_done global_id 4104 payload 824 2023-09-30T00:44:13.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a74003ab0 msgr2=0x7f1a8c141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a74003ab0 0x7f1a8c141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/430427971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a8c098a40 msgr2=0x7f1a8c141ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a8c098a40 0x7f1a8c141ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 monclient: found mon.noname-a 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.379+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/430427971 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a8c11bbe0 con 0x7f1a74004e70 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a8b7fe700 1 -- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 msgr2=0x7f1a8c141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a8b7fe700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a8c141990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a80006db0 tx=0x7f1a80008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:13.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a8000c3b0 con 0x7f1a74004e70 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a73fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a73fff700 10 monclient: dump: 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:13.398 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a73fff700 10 monclient: _finish_auth 0 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.381+0000 7f1a73fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 5 monclient: authenticate success, global_id 4104 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a8001c070 con 0x7f1a74004e70 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 10 monclient: _renew_subs 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a8c0a1500 con 0x7f1a74004e70 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a80018450 con 0x7f1a74004e70 2023-09-30T00:44:13.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 10 monclient: dump: 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:13.400 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 10 monclient: _renew_subs 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a8a7fc700 4 set_mon_vals no callback set 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a8c12e6a0 con 0x7f1a74004e70 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1a8001e030 con 0x7f1a74004e70 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.382+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1a80021780 con 0x7f1a74004e70 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.383+0000 7f1a925d9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.383+0000 7f1a925d9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.383+0000 7f1a925d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.383+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a8c072b00 con 0x7f1a74004e70 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.385+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1a8000f040 con 0x7f1a74004e70 2023-09-30T00:44:13.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.385+0000 7f1a73fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:13.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.385+0000 7f1a73fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.466+0000 7f1a925d9700 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T00:44:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.466+0000 7f1a925d9700 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T00:44:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.466+0000 7f1a925d9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.466+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f1a8c063920 con 0x7f1a74004e70 2023-09-30T00:44:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.470+0000 7f1a73fff700 1 -- 172.21.15.6:0/430427971 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1a80018ce0 con 0x7f1a74004e70 2023-09-30T00:44:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.470+0000 7f1a73fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2023-09-30T00:44:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.470+0000 7f1a73fff700 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2023-09-30T00:44:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 10 monclient: shutdown 2023-09-30T00:44:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 msgr2=0x7f1a8c141990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a8c141990 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1a80006db0 tx=0x7f1a80008040 comp rx=0 tx=0).stop 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 shutdown_connections 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1a8c098a40 0x7f1a8c141ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1a74003ab0 0x7f1a8c141450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 --2- 172.21.15.6:0/430427971 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1a74004e70 0x7f1a8c141990 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 >> 172.21.15.6:0/430427971 conn(0x7f1a8c0a5480 msgr2=0x7f1a8c1308d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 shutdown_connections 2023-09-30T00:44:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:13.471+0000 7f1a925d9700 1 -- 172.21.15.6:0/430427971 wait complete. 2023-09-30T00:44:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:adjusted tunables profile to default 2023-09-30T00:44:13.485 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2023-09-30T00:44:13.486 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:44:13.486 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2023-09-30T00:44:13.517 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:44:13.517 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2023-09-30T00:44:13.523 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:13.523+0000 7f3cbdc3d1c0 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-09-30T00:44:13.590 DEBUG:teuthology.orchestra.run.smithi006:> set -ex 2023-09-30T00:44:13.590 DEBUG:teuthology.orchestra.run.smithi006:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2023-09-30T00:44:13.665 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:13.665 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2023-09-30T00:44:13.715 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:13.716 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2023-09-30T00:44:13.719 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:13.718+0000 7f3cbdc3d1c0 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-09-30T00:44:13.752 DEBUG:teuthology.orchestra.run.smithi022:> set -ex 2023-09-30T00:44:13.753 DEBUG:teuthology.orchestra.run.smithi022:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2023-09-30T00:44:13.822 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 97153c5d-1a02-4a71-82e1-ab2dc690ab09 0 2023-09-30T00:44:13.991 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.991+0000 7fc8df45e700 10 monclient: get_monmap_and_config 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.991+0000 7fc8df45e700 10 monclient: build_initial_monmap 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.991+0000 7fc8df45e700 10 monclient: monmap: 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:44:13.991951+0000 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:44:13.991951+0000 2023-09-30T00:44:13.992 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:13.993 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.991+0000 7fc8df45e700 1 Processor -- start 2023-09-30T00:44:13.998 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 -- start start 2023-09-30T00:44:13.998 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient: init 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136a80 0x7fc8d8136e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): picked mon.noname-b con 0x7fc8d8136a80 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): picked mon.noname-c con 0x7fc8d8136600 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc8d8098a90 0x7fc8d80a0330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): picked mon.noname-a con 0x7fc8d8098a90 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d8097e50 con 0x7fc8d8098a90 2023-09-30T00:44:13.999 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8d8098130 con 0x7fc8d8136600 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8097b70 con 0x7fc8d8136a80 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8df45e700 10 monclient(hunting): authenticate will time out at 1129.301684s 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38358/0 (socket says 172.21.15.22:38358) 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 1 -- 172.21.15.22:0/2247930684 learned_addr learned my addr 172.21.15.22:0/2247930684 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 10 monclient(hunting): get_auth_request con 0x7fc8d8136600 auth_method 0 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.000 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 1 -- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 msgr2=0x7fc8d809fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 1 -- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 msgr2=0x7fc8d809fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.992+0000 7fc8dc9f9700 1 --2- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dc9f9700 1 --2- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136a80 0x7fc8d8136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/2247930684 >> v2:172.21.15.6:3300/0 conn(0x7fc8d8098a90 0x7fc8d80a0330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): get_auth_request con 0x7fc8d8136a80 auth_method 0 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): get_auth_request con 0x7fc8d8098a90 auth_method 0 2023-09-30T00:44:14.001 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_done global_id 4107 payload 274 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136a80 msgr2=0x7fc8d8136e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:14.002 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136a80 0x7fc8d8136e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 msgr2=0x7fc8d809fdf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient: _finish_hunting 0 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 monclient: found mon.noname-a 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/2247930684 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d8137390 con 0x7fc8d8098a90 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.993+0000 7fc8dd1fa700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136a80 0x7fc8d8136e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 msgr2=0x7fc8d80a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d80a0330 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc8d806a3e0 tx=0x7fc8d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2afa4ffe034ac8ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:14.003 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 1 -- 172.21.15.22:0/2247930684 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0011840 con 0x7fc8d8098a90 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 10 monclient: dump: 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.004 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 10 monclient: _finish_auth 0 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 1 -- 172.21.15.22:0/2247930684 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d00119a0 con 0x7fc8d8098a90 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8cffff700 1 -- 172.21.15.22:0/2247930684 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0011d40 con 0x7fc8d8098a90 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.995+0000 7fc8df45e700 5 monclient: authenticate success, global_id 4107 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 4 set_mon_vals no callback set 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8cffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8cffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:14.005 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8cffff700 10 monclient: dump: 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 10 monclient: shutdown 2023-09-30T00:44:14.006 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 -- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 msgr2=0x7fc8d80a0330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d80a0330 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc8d806a3e0 tx=0x7fc8d00091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 -- 172.21.15.22:0/2247930684 shutdown_connections 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d80a0330 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 --2- 172.21.15.22:0/2247930684 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136a80 0x7fc8d8136e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 --2- 172.21.15.22:0/2247930684 >> v2:172.21.15.22:3300/0 conn(0x7fc8d8136600 0x7fc8d809fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 -- 172.21.15.22:0/2247930684 >> 172.21.15.22:0/2247930684 conn(0x7fc8d80a1ef0 msgr2=0x7fc8d80a22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 -- 172.21.15.22:0/2247930684 shutdown_connections 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.996+0000 7fc8df45e700 1 -- 172.21.15.22:0/2247930684 wait complete. 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.997+0000 7fc8df45e700 10 monclient: build_initial_monmap 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.997+0000 7fc8df45e700 10 monclient: monmap: 2023-09-30T00:44:14.007 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.998+0000 7fc8df45e700 1 Processor -- start 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.998+0000 7fc8df45e700 1 -- start start 2023-09-30T00:44:14.008 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.998+0000 7fc8df45e700 10 monclient: init 2023-09-30T00:44:14.013 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:14.013 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): picked mon.noname-a con 0x7fc8d8098a90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136600 0x7fc8d806ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): picked mon.noname-c con 0x7fc8d8136600 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 0x7fc8d806b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): picked mon.noname-b con 0x7fc8d8136a80 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d8096e80 con 0x7fc8d8098a90 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.014 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d80a4dc0 con 0x7fc8d8136a80 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8d806dbe0 con 0x7fc8d8136600 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8df45e700 10 monclient(hunting): authenticate will time out at 1129.308287s 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 0x7fc8d806b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 0x7fc8d806b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38366/0 (socket says 172.21.15.22:38366) 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/3149680282 learned_addr learned my addr 172.21.15.22:0/3149680282 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): get_auth_request con 0x7fc8d8136a80 auth_method 0 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.015 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136600 0x7fc8d806ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): get_auth_request con 0x7fc8d8098a90 auth_method 0 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): get_auth_request con 0x7fc8d8136600 auth_method 0 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.016 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd1fa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.017 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dc9f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient(hunting): handle_auth_done global_id 4108 payload 824 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136600 msgr2=0x7fc8d806ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136600 0x7fc8d806ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 msgr2=0x7fc8d806a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d806a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient: _finish_hunting 0 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 monclient: found mon.noname-b 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:14.018 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:13.999+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/3149680282 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8d4009a20 con 0x7fc8d8136a80 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8dd9fb700 1 -- 172.21.15.22:0/3149680282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 msgr2=0x7fc8d806b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8dd9fb700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 0x7fc8d806b390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000b930 tx=0x7fc8d0017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0006af0 con 0x7fc8d8136a80 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: dump: 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.019 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: _finish_auth 0 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:14.020 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8d0024440 con 0x7fc8d8136a80 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 5 monclient: authenticate success, global_id 4108 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 10 monclient: _renew_subs 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8d0009420 con 0x7fc8d8136a80 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8d0024800 con 0x7fc8d8136a80 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 10 monclient: dump: 2023-09-30T00:44:14.021 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cf7fe700 4 set_mon_vals no callback set 2023-09-30T00:44:14.022 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 10 monclient: _renew_subs 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8d806ccf0 con 0x7fc8d8136a80 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fc8d001a6a0 con 0x7fc8d8136a80 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.002+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc8d002d740 con 0x7fc8d8136a80 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.003+0000 7fc8df45e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.003+0000 7fc8df45e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.003+0000 7fc8df45e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.003+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8d8072b00 con 0x7fc8d8136a80 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.005+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fc8d0015020 con 0x7fc8d8136a80 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.005+0000 7fc8cd7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.023 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.005+0000 7fc8cd7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:14.094 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.093+0000 7fc8df45e700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "97153c5d-1a02-4a71-82e1-ab2dc690ab09", "id": 0}] 2023-09-30T00:44:14.094 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.093+0000 7fc8df45e700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "97153c5d-1a02-4a71-82e1-ab2dc690ab09", "id": 0}] 2023-09-30T00:44:14.094 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.093+0000 7fc8df45e700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.094 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.093+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd new", "uuid": "97153c5d-1a02-4a71-82e1-ab2dc690ab09", "id": 0} v 0) v1 -- 0x7fc8d806de20 con 0x7fc8d8136a80 2023-09-30T00:44:14.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.097+0000 7fc8cd7fa700 1 -- 172.21.15.22:0/3149680282 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "97153c5d-1a02-4a71-82e1-ab2dc690ab09", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7fc8d001aae0 con 0x7fc8d8136a80 2023-09-30T00:44:14.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.097+0000 7fc8cd7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "97153c5d-1a02-4a71-82e1-ab2dc690ab09", "id": 0}] 2023-09-30T00:44:14.098 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.097+0000 7fc8cd7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:14.098 INFO:teuthology.orchestra.run.smithi022.stdout:0 2023-09-30T00:44:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 10 monclient: shutdown 2023-09-30T00:44:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 msgr2=0x7fc8d806b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 0x7fc8d806b390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fc8d000b930 tx=0x7fc8d0017000 comp rx=0 tx=0).stop 2023-09-30T00:44:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 shutdown_connections 2023-09-30T00:44:14.099 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc8d8136a80 0x7fc8d806b390 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc8d8098a90 0x7fc8d806a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 --2- 172.21.15.22:0/3149680282 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc8d8136600 0x7fc8d806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.098+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 >> 172.21.15.22:0/3149680282 conn(0x7fc8d80a1ef0 msgr2=0x7fc8d80a3b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:14.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.099+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 shutdown_connections 2023-09-30T00:44:14.100 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.099+0000 7fc8df45e700 1 -- 172.21.15.22:0/3149680282 wait complete. 2023-09-30T00:44:14.101 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:14.101+0000 7f3cbdc3d1c0 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-09-30T00:44:14.112 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d 1 2023-09-30T00:44:14.242 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.242+0000 7f9ce9bb7700 10 monclient: get_monmap_and_config 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.242+0000 7f9ce9bb7700 10 monclient: build_initial_monmap 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.242+0000 7f9ce9bb7700 10 monclient: monmap: 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:44:14.243274+0000 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:44:14.243274+0000 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:14.243 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.244 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:14.244 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:14.244 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:14.244 INFO:teuthology.orchestra.run.smithi022.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:14.244 INFO:teuthology.orchestra.run.smithi022.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:14.244 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.249 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 Processor -- start 2023-09-30T00:44:14.249 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 -- start start 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient: init 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient: _add_conns ranks=[2,3,0,1,4] 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 0x7f9ce4132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): picked mon.noname-b con 0x7f9ce4098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ce4098a40 0x7f9ce412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): picked mon.noname-a con 0x7f9ce4098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9ce4098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:14.250 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ce40a1000 con 0x7f9ce4098e90 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ce40a0a50 con 0x7f9ce4098a40 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce40a0730 con 0x7f9ce4098670 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.243+0000 7f9ce9bb7700 10 monclient(hunting): authenticate will time out at 1129.552998s 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 0x7f9ce4132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.251 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ce4098a40 0x7f9ce412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9ce4098a40 0x7f9ce412ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.22:38770/0 (socket says 172.21.15.22:38770) 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 0x7f9ce4132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.22:40092/0 (socket says 172.21.15.22:40092) 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 -- 172.21.15.22:0/3354902117 learned_addr learned my addr 172.21.15.22:0/3354902117 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): get_auth_request con 0x7f9ce4098a40 auth_method 0 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): get_auth_request con 0x7f9ce4098670 auth_method 0 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.252 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_done global_id 4113 payload 274 2023-09-30T00:44:14.253 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 -- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 msgr2=0x7f9ce4132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 --2- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 0x7f9ce4132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 -- 172.21.15.22:0/3354902117 >> v1:172.21.15.6:6789/0 conn(0x7f9ce4098e90 legacy=0x7f9ce412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 monclient: found mon.noname-a 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce2ffd700 1 -- 172.21.15.22:0/3354902117 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce4068df0 con 0x7f9ce4098a40 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.244+0000 7f9ce37fe700 1 --2- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 0x7f9ce4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:14.254 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce2ffd700 1 -- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 msgr2=0x7f9ce412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce2ffd700 1 --2- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce412ede0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80099b0 tx=0x7f9cd8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e597733cddcd449 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 1 -- 172.21.15.22:0/3354902117 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd8009ea0 con 0x7f9ce4098a40 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: dump: 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.255 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:14.256 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 5 monclient: authenticate success, global_id 4113 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 1 -- 172.21.15.22:0/3354902117 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd800ba30 con 0x7f9ce4098a40 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 4 set_mon_vals no callback set 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 1 -- 172.21.15.22:0/3354902117 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd800bdf0 con 0x7f9ce4098a40 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:14.257 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce27fc700 10 monclient: dump: 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.258 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 10 monclient: shutdown 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 msgr2=0x7f9ce412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce412ede0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f9cd80099b0 tx=0x7f9cd8009250 comp rx=0 tx=0).stop 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/3354902117 shutdown_connections 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098670 0x7f9ce4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.247+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/3354902117 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce412ede0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.248+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/3354902117 >> 172.21.15.22:0/3354902117 conn(0x7f9ce40a5480 msgr2=0x7f9ce40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:14.259 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.248+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/3354902117 shutdown_connections 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.248+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/3354902117 wait complete. 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.248+0000 7f9ce9bb7700 10 monclient: build_initial_monmap 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.248+0000 7f9ce9bb7700 10 monclient: monmap: 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:14.260 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.249+0000 7f9ce9bb7700 1 Processor -- start 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.249+0000 7f9ce9bb7700 1 -- start start 2023-09-30T00:44:14.261 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.249+0000 7f9ce9bb7700 10 monclient: init 2023-09-30T00:44:14.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:14.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:14.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 0x7f9ce4141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): picked mon.noname-b con 0x7f9ce4098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:14.266 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce4141f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): picked mon.noname-a con 0x7f9ce4098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098e90 0x7f9ce409a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): picked mon.noname-c con 0x7f9ce4098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4135390 con 0x7f9ce4098a40 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4099d40 con 0x7f9ce4098670 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4099a60 con 0x7f9ce4098e90 2023-09-30T00:44:14.267 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce9bb7700 10 monclient(hunting): authenticate will time out at 1129.559471s 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 0x7f9ce4141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 0x7f9ce4141950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38370/0 (socket says 172.21.15.22:38370) 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 1 -- 172.21.15.22:0/1154000626 learned_addr learned my addr 172.21.15.22:0/1154000626 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): get_auth_request con 0x7f9ce4098670 auth_method 0 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.268 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce4141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): get_auth_request con 0x7f9ce4098a40 auth_method 0 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098e90 0x7f9ce409a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.269 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): get_auth_request con 0x7f9ce4098e90 auth_method 0 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.270 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 10 monclient(hunting): handle_auth_done global_id 4111 payload 824 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 1 -- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098e90 msgr2=0x7f9ce409a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.250+0000 7f9ce37fe700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098e90 0x7f9ce409a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 1 -- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 msgr2=0x7f9ce4141f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce4141f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 1 monclient: found mon.noname-b 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:14.271 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.251+0000 7f9ce37fe700 1 -- 172.21.15.22:0/1154000626 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4009420 con 0x7f9ce4098670 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce37fe700 1 -- 172.21.15.22:0/1154000626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 msgr2=0x7f9ce4141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce37fe700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 0x7f9ce4141950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400ecb0 tx=0x7f9cd40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400cb30 con 0x7f9ce4098670 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: dump: 2023-09-30T00:44:14.272 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.273 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd400cc90 con 0x7f9ce4098670 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 5 monclient: authenticate success, global_id 4111 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 10 monclient: _renew_subs 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cd80094f0 con 0x7f9ce4098670 2023-09-30T00:44:14.274 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd4013470 con 0x7f9ce4098670 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 10 monclient: dump: 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.275 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 10 monclient: _renew_subs 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4099800 con 0x7f9ce4098670 2023-09-30T00:44:14.276 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ce1ffb700 4 set_mon_vals no callback set 2023-09-30T00:44:14.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f9cd4005c80 con 0x7f9ce4098670 2023-09-30T00:44:14.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.253+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 836+0+0 (secure 0 0 0) 0x7f9cd40063e0 con 0x7f9ce4098670 2023-09-30T00:44:14.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.254+0000 7f9ce9bb7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.254+0000 7f9ce9bb7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.254+0000 7f9ce9bb7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.277 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.254+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4099eb0 con 0x7f9ce4098670 2023-09-30T00:44:14.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.256+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f9cd4021060 con 0x7f9ce4098670 2023-09-30T00:44:14.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.256+0000 7f9ccb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.278 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.256+0000 7f9ccb7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.345+0000 7f9ce9bb7700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d", "id": 1}] 2023-09-30T00:44:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.345+0000 7f9ce9bb7700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d", "id": 1}] 2023-09-30T00:44:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.345+0000 7f9ce9bb7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.345+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd new", "uuid": "7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d", "id": 1} v 0) v1 -- 0x7f9ce4072b00 con 0x7f9ce4098670 2023-09-30T00:44:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.349+0000 7f9ccb7fe700 1 -- 172.21.15.22:0/1154000626 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7f9cd4017070 con 0x7f9ce4098670 2023-09-30T00:44:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.349+0000 7f9ccb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d", "id": 1}] 2023-09-30T00:44:14.349 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.349+0000 7f9ccb7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:14.350 INFO:teuthology.orchestra.run.smithi022.stdout:1 2023-09-30T00:44:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 10 monclient: shutdown 2023-09-30T00:44:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 msgr2=0x7f9ce4141950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 0x7f9ce4141950 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f9cd400ecb0 tx=0x7f9cd40091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 shutdown_connections 2023-09-30T00:44:14.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9ce4098e90 0x7f9ce409a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9ce4098670 0x7f9ce4141950 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 --2- 172.21.15.22:0/1154000626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9ce4098a40 0x7f9ce4141f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 >> 172.21.15.22:0/1154000626 conn(0x7f9ce40a5480 msgr2=0x7f9ce41355a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 shutdown_connections 2023-09-30T00:44:14.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.350+0000 7f9ce9bb7700 1 -- 172.21.15.22:0/1154000626 wait complete. 2023-09-30T00:44:14.363 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee 2 2023-09-30T00:44:14.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.494+0000 7f1e40504700 10 monclient: get_monmap_and_config 2023-09-30T00:44:14.494 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.494+0000 7f1e40504700 10 monclient: build_initial_monmap 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.494+0000 7f1e40504700 10 monclient: monmap: 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:44:14.495078+0000 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:44:14.495078+0000 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:14.495 INFO:teuthology.orchestra.run.smithi022.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:14.496 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.500 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.494+0000 7f1e40504700 1 Processor -- start 2023-09-30T00:44:14.500 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 1 -- start start 2023-09-30T00:44:14.500 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient: init 2023-09-30T00:44:14.500 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e38132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): picked mon.noname-b con 0x7f1e38098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1e38098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1e38098e90 0x7f1e3812f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): picked mon.noname-c con 0x7f1e38098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e380a1000 con 0x7f1e38098a40 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e380a0a50 con 0x7f1e38098e90 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.501 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380a0730 con 0x7f1e38098670 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e40504700 10 monclient(hunting): authenticate will time out at 1129.804852s 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.495+0000 7f1e3eaa1700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1e38098e90 0x7f1e3812f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3da9f700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f1e38098a40 0x7f1e3812ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:40308/0 (socket says 172.21.15.22:40308) 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3da9f700 1 -- 172.21.15.22:0/2905138706 learned_addr learned my addr 172.21.15.22:0/2905138706 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): get_auth_request con 0x7f1e38098e90 auth_method 0 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.502 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 1 --2- 172.21.15.22:0/2905138706 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e38132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): get_auth_request con 0x7f1e38098670 auth_method 0 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3e2a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_done global_id 4114 payload 274 2023-09-30T00:44:14.503 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/2905138706 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 msgr2=0x7f1e38132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 1 --2- 172.21.15.22:0/2905138706 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e38132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/2905138706 >> v1:172.21.15.22:6789/0 conn(0x7f1e38098a40 legacy=0x7f1e3812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient: _finish_hunting 0 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 1 monclient: found mon.noname-c 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.496+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/2905138706 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e38068df0 con 0x7f1e38098e90 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/2905138706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 msgr2=0x7f1e3812f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3eaa1700 1 --2- 172.21.15.22:0/2905138706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e3812f4f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e300098e0 tx=0x7f1e300091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b51c72f3ea70c1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 1 -- 172.21.15.22:0/2905138706 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30011840 con 0x7f1e38098e90 2023-09-30T00:44:14.504 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: dump: 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.505 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: _finish_auth 0 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e40504700 5 monclient: authenticate success, global_id 4114 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 1 -- 172.21.15.22:0/2905138706 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e300119a0 con 0x7f1e38098e90 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e40504700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e40504700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e40504700 4 set_mon_vals no callback set 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 1 -- 172.21.15.22:0/2905138706 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e30011d40 con 0x7f1e38098e90 2023-09-30T00:44:14.506 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.498+0000 7f1e3d29e700 10 monclient: dump: 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.507 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 10 monclient: shutdown 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 -- 172.21.15.22:0/2905138706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 msgr2=0x7f1e3812f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 --2- 172.21.15.22:0/2905138706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e3812f4f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f1e300098e0 tx=0x7f1e300091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 -- 172.21.15.22:0/2905138706 shutdown_connections 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 --2- 172.21.15.22:0/2905138706 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e3812f4f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 --2- 172.21.15.22:0/2905138706 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e38132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 -- 172.21.15.22:0/2905138706 >> 172.21.15.22:0/2905138706 conn(0x7f1e380a5480 msgr2=0x7f1e380a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 -- 172.21.15.22:0/2905138706 shutdown_connections 2023-09-30T00:44:14.508 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.499+0000 7f1e40504700 1 -- 172.21.15.22:0/2905138706 wait complete. 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.500+0000 7f1e40504700 10 monclient: build_initial_monmap 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.500+0000 7f1e40504700 10 monclient: monmap: 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:14.509 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:14.510 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.500+0000 7f1e40504700 1 Processor -- start 2023-09-30T00:44:14.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.500+0000 7f1e40504700 1 -- start start 2023-09-30T00:44:14.510 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.500+0000 7f1e40504700 10 monclient: init 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e3812be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): picked mon.noname-c con 0x7f1e38098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e38098a40 0x7f1e3812c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): picked mon.noname-a con 0x7f1e38098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e38099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): picked mon.noname-b con 0x7f1e38098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.514 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3809d2b0 con 0x7f1e38098a40 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380a3bb0 con 0x7f1e38098e90 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e380a1500 con 0x7f1e38098670 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e40504700 10 monclient(hunting): authenticate will time out at 1129.810106s 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e38099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e3812be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e38099290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38394/0 (socket says 172.21.15.22:38394) 2023-09-30T00:44:14.515 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/647632182 learned_addr learned my addr 172.21.15.22:0/647632182 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): get_auth_request con 0x7f1e38098e90 auth_method 0 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): get_auth_request con 0x7f1e38098670 auth_method 0 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e38098a40 0x7f1e3812c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.517 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): get_auth_request con 0x7f1e38098a40 auth_method 0 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3e2a0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:14.518 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3da9f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient(hunting): handle_auth_done global_id 4117 payload 824 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 msgr2=0x7f1e3812be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e3812be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e38098a40 msgr2=0x7f1e3812c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e38098a40 0x7f1e3812c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient: _finish_hunting 0 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 monclient: found mon.noname-b 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.501+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/647632182 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e30009420 con 0x7f1e38098e90 2023-09-30T00:44:14.519 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.503+0000 7f1e3eaa1700 1 -- 172.21.15.22:0/647632182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 msgr2=0x7f1e38099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.503+0000 7f1e3eaa1700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e38099290 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1e300093f0 tx=0x7f1e30016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3000bd00 con 0x7f1e38098e90 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: dump: 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.520 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e30020070 con 0x7f1e38098e90 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e3001b6a0 con 0x7f1e38098e90 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:14.521 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 10 monclient: dump: 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:14.522 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 5 monclient: authenticate success, global_id 4117 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e3ca9d700 4 set_mon_vals no callback set 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 10 monclient: _renew_subs 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e3809b270 con 0x7f1e38098e90 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 10 monclient: _renew_subs 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e3809b930 con 0x7f1e38098e90 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1e30022030 con 0x7f1e38098e90 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.504+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 951+0+0 (secure 0 0 0) 0x7f1e3001c880 con 0x7f1e38098e90 2023-09-30T00:44:14.523 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.505+0000 7f1e40504700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.505+0000 7f1e40504700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.505+0000 7f1e40504700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.505+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3809d110 con 0x7f1e38098e90 2023-09-30T00:44:14.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.507+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1e3809d110 con 0x7f1e38098e90 2023-09-30T00:44:14.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.507+0000 7f1e367fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:14.524 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.507+0000 7f1e367fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:14.597 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.597+0000 7f1e40504700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee", "id": 2}] 2023-09-30T00:44:14.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.597+0000 7f1e40504700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee", "id": 2}] 2023-09-30T00:44:14.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.597+0000 7f1e40504700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:14.598 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:14.597+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd new", "uuid": "6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee", "id": 2} v 0) v1 -- 0x7f1e3809cd80 con 0x7f1e38098e90 2023-09-30T00:44:14.693 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:14.693+0000 7f3cbdc3d1c0 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-09-30T00:44:14.847 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:14.847+0000 7f3cbdc3d1c0 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-09-30T00:44:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.212+0000 7f1e367fc700 1 -- 172.21.15.22:0/647632182 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f1e3001c450 con 0x7f1e38098e90 2023-09-30T00:44:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.212+0000 7f1e367fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee", "id": 2}] 2023-09-30T00:44:15.212 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.212+0000 7f1e367fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:15.213 INFO:teuthology.orchestra.run.smithi022.stdout:2 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 10 monclient: shutdown 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 msgr2=0x7f1e38099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e38099290 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1e300093f0 tx=0x7f1e30016000 comp rx=0 tx=0).stop 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 shutdown_connections 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1e38098e90 0x7f1e38099290 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1e38098670 0x7f1e3812be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 --2- 172.21.15.22:0/647632182 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1e38098a40 0x7f1e3812c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.214 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 >> 172.21.15.22:0/647632182 conn(0x7f1e380a5480 msgr2=0x7f1e38069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 shutdown_connections 2023-09-30T00:44:15.215 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.213+0000 7f1e40504700 1 -- 172.21.15.22:0/647632182 wait complete. 2023-09-30T00:44:15.226 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new f0be84e5-ee32-4470-9d79-7c4942bae02b 3 2023-09-30T00:44:15.239 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:15.239+0000 7f3cbdc3d1c0 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-09-30T00:44:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.357+0000 7fb72c9b7700 10 monclient: get_monmap_and_config 2023-09-30T00:44:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.357+0000 7fb72c9b7700 10 monclient: build_initial_monmap 2023-09-30T00:44:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.357+0000 7fb72c9b7700 10 monclient: monmap: 2023-09-30T00:44:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.358 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:44:15.358601+0000 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:44:15.358601+0000 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:15.359 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.358+0000 7fb72c9b7700 1 Processor -- start 2023-09-30T00:44:15.361 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.358+0000 7fb72c9b7700 1 -- start start 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.358+0000 7fb72c9b7700 10 monclient: init 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fb728098670 0x7fb728132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient(hunting): picked mon.noname-c con 0x7fb728098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fb728098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb728098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7280a1000 con 0x7fb728098a40 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb72812fc00 con 0x7fb728098e90 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.362 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7280a0a50 con 0x7fb728098670 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72c9b7700 10 monclient(hunting): authenticate will time out at 1130.668284s 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fb728098670 0x7fb728132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb726d9d700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fb728098e90 0x7fb72812f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.22:40310/0 (socket says 172.21.15.22:40310) 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb726d9d700 1 -- 172.21.15.22:0/3690643567 learned_addr learned my addr 172.21.15.22:0/3690643567 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): get_auth_request con 0x7fb728098670 auth_method 0 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.363 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient(hunting): handle_auth_done global_id 4120 payload 274 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 1 -- 172.21.15.22:0/3690643567 >> v1:172.21.15.22:6789/0 conn(0x7fb728098e90 legacy=0x7fb72812f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 1 -- 172.21.15.22:0/3690643567 >> v1:172.21.15.6:6789/0 conn(0x7fb728098a40 legacy=0x7fb72812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient: _finish_hunting 0 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 1 monclient: found mon.noname-c 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:15.364 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.359+0000 7fb72659c700 1 -- 172.21.15.22:0/3690643567 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7280a0730 con 0x7fb728098670 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72659c700 1 -- 172.21.15.22:0/3690643567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 msgr2=0x7fb728132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72659c700 1 --2- 172.21.15.22:0/3690643567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb728132d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb7180098e0 tx=0x7fb7180091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20eaf148bd627d85 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 1 -- 172.21.15.22:0/3690643567 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71800b9f0 con 0x7fb728098670 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: dump: 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.369 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: _finish_auth 0 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 1 -- 172.21.15.22:0/3690643567 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb71800bb50 con 0x7fb728098670 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 5 monclient: authenticate success, global_id 4120 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:15.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 1 -- 172.21.15.22:0/3690643567 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb718011420 con 0x7fb728098670 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 4 set_mon_vals no callback set 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72559a700 10 monclient: dump: 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.371 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 10 monclient: shutdown 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 1 -- 172.21.15.22:0/3690643567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 msgr2=0x7fb728132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 1 --2- 172.21.15.22:0/3690643567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb728132d00 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb7180098e0 tx=0x7fb7180091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 1 -- 172.21.15.22:0/3690643567 shutdown_connections 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 1 --2- 172.21.15.22:0/3690643567 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb728132d00 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.362+0000 7fb72c9b7700 1 -- 172.21.15.22:0/3690643567 >> 172.21.15.22:0/3690643567 conn(0x7fb7280a5480 msgr2=0x7fb7280a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.363+0000 7fb72c9b7700 1 -- 172.21.15.22:0/3690643567 shutdown_connections 2023-09-30T00:44:15.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.363+0000 7fb72c9b7700 1 -- 172.21.15.22:0/3690643567 wait complete. 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.363+0000 7fb72c9b7700 10 monclient: build_initial_monmap 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.363+0000 7fb72c9b7700 10 monclient: monmap: 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:15.373 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.364+0000 7fb72c9b7700 1 Processor -- start 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.364+0000 7fb72c9b7700 1 -- start start 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.364+0000 7fb72c9b7700 10 monclient: init 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb72812c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): picked mon.noname-b con 0x7fb728098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb728098a40 0x7fb7281417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): picked mon.noname-c con 0x7fb728098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb728098e90 0x7fb72809cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): picked mon.noname-a con 0x7fb728098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7280a3af0 con 0x7fb728098e90 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72811c8f0 con 0x7fb728098670 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb72809f210 con 0x7fb728098a40 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72c9b7700 10 monclient(hunting): authenticate will time out at 1130.674285s 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb72812c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb72812c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38416/0 (socket says 172.21.15.22:38416) 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 -- 172.21.15.22:0/511966879 learned_addr learned my addr 172.21.15.22:0/511966879 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:15.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): get_auth_request con 0x7fb728098670 auth_method 0 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb726d9d700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb728098e90 0x7fb72809cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb725d9b700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb728098a40 0x7fb7281417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient(hunting): handle_auth_done global_id 4123 payload 824 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 -- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb728098a40 msgr2=0x7fb7281417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb728098a40 0x7fb7281417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 -- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb728098e90 msgr2=0x7fb72809cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb728098e90 0x7fb72809cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient: _finish_hunting 0 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 monclient: found mon.noname-b 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb72659c700 1 -- 172.21.15.22:0/511966879 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb718009420 con 0x7fb728098670 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb725d9b700 10 monclient: get_auth_request con 0x7fb728098a40 auth_method 0 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb726d9d700 10 monclient: get_auth_request con 0x7fb728098e90 auth_method 0 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb725d9b700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb728098a40 0x7fb7281417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.365+0000 7fb726d9d700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb728098e90 0x7fb72809cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.367+0000 7fb72659c700 1 -- 172.21.15.22:0/511966879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 msgr2=0x7fb72812c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:15.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.367+0000 7fb72659c700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb72812c150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb71800b3c0 tx=0x7fb718016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:15.378 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.367+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 DispatchQueue::entry inject delay of 0.002000 2023-09-30T00:44:15.379 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.369+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71801b450 con 0x7fb728098670 2023-09-30T00:44:15.379 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.379 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: dump: 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.380 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb718020070 con 0x7fb728098670 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 5 monclient: authenticate success, global_id 4123 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb71801b7e0 con 0x7fb728098670 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.381 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 10 monclient: dump: 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.382 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 10 monclient: _renew_subs 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb724d99700 4 set_mon_vals no callback set 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72811c2a0 con 0x7fb728098670 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 10 monclient: _renew_subs 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72809e5f0 con 0x7fb728098670 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fb718022030 con 0x7fb728098670 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.370+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1066+0+0 (secure 0 0 0) 0x7fb71801cb20 con 0x7fb728098670 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.371+0000 7fb72c9b7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:15.383 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.371+0000 7fb72c9b7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.371+0000 7fb72c9b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.371+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72809e5b0 con 0x7fb728098670 2023-09-30T00:44:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.373+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fb71801cdc0 con 0x7fb728098670 2023-09-30T00:44:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.373+0000 7fb7167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:15.384 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.373+0000 7fb7167fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:15.462 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.462+0000 7fb72c9b7700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f0be84e5-ee32-4470-9d79-7c4942bae02b", "id": 3}] 2023-09-30T00:44:15.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.462+0000 7fb72c9b7700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f0be84e5-ee32-4470-9d79-7c4942bae02b", "id": 3}] 2023-09-30T00:44:15.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.462+0000 7fb72c9b7700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:15.463 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.462+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd new", "uuid": "f0be84e5-ee32-4470-9d79-7c4942bae02b", "id": 3} v 0) v1 -- 0x7fb72809e5b0 con 0x7fb728098670 2023-09-30T00:44:15.465 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.465+0000 7fb7167fc700 1 -- 172.21.15.22:0/511966879 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f0be84e5-ee32-4470-9d79-7c4942bae02b", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fb71801c680 con 0x7fb728098670 2023-09-30T00:44:15.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.465+0000 7fb7167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f0be84e5-ee32-4470-9d79-7c4942bae02b", "id": 3}] 2023-09-30T00:44:15.466 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.465+0000 7fb7167fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:15.466 INFO:teuthology.orchestra.run.smithi022.stdout:3 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 10 monclient: shutdown 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 msgr2=0x7fb72812c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb72812c150 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb71800b3c0 tx=0x7fb718016000 comp rx=0 tx=0).stop 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 shutdown_connections 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb728098e90 0x7fb72809cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.467 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb728098670 0x7fb72812c150 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 --2- 172.21.15.22:0/511966879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb728098a40 0x7fb7281417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 >> 172.21.15.22:0/511966879 conn(0x7fb7280a5480 msgr2=0x7fb72812f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:15.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 shutdown_connections 2023-09-30T00:44:15.468 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.466+0000 7fb72c9b7700 1 -- 172.21.15.22:0/511966879 wait complete. 2023-09-30T00:44:15.479 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 061ce3a2-0e9b-448e-9cae-789937c85eab 4 2023-09-30T00:44:15.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.649+0000 7f03a57de700 10 monclient: get_monmap_and_config 2023-09-30T00:44:15.649 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.649+0000 7f03a57de700 10 monclient: build_initial_monmap 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.649+0000 7f03a57de700 10 monclient: monmap: 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:44:15.650096+0000 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:44:15.650096+0000 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:15.650 INFO:teuthology.orchestra.run.smithi022.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:15.651 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.649+0000 7f03a57de700 1 Processor -- start 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 1 -- start start 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient: init 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03a0132150 0x7f03a0132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): picked mon.noname-a con 0x7f03a0132150 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:15.655 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132e10 0x7f03a0098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): picked mon.noname-b con 0x7f03a0132e10 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03a0098ca0 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03a0132520 con 0x7f03a0098ca0 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03a0099120 con 0x7f03a0132150 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a0096a80 con 0x7f03a0132e10 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03a57de700 10 monclient(hunting): authenticate will time out at 1130.959680s 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03a0132150 0x7f03a0132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03967fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132e10 0x7f03a0098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.656 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039f7fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03a0098ca0 0x7f03a0068df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.22:52314/0 (socket says 172.21.15.22:52314) 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039f7fe700 1 -- 172.21.15.22:0/668184648 learned_addr learned my addr 172.21.15.22:0/668184648 (peer_addr_for_me v1:172.21.15.22:0/0) 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039effd700 10 monclient(hunting): get_auth_request con 0x7f03a0132150 auth_method 0 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f039effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03967fc700 10 monclient(hunting): get_auth_request con 0x7f03a0132e10 auth_method 0 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03967fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.650+0000 7f03967fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.657 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f03967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f03967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f03967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 10 monclient(hunting): handle_auth_done global_id 4122 payload 274 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 1 -- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132e10 msgr2=0x7f03a0098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 1 --2- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132e10 0x7f03a0098760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 1 -- 172.21.15.22:0/668184648 >> v1:172.21.15.6:6789/0 conn(0x7f03a0098ca0 legacy=0x7f03a0068df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 1 monclient: found mon.noname-a 2023-09-30T00:44:15.658 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f039effd700 1 -- 172.21.15.22:0/668184648 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a00a0f80 con 0x7f03a0132150 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.651+0000 7f03967fc700 1 --2- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132e10 0x7f03a0098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039effd700 1 -- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132150 msgr2=0x7f03a0132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039effd700 1 --2- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132150 0x7f03a0132790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0390009910 tx=0x7f03900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e23e3aeb3e3d529 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 1 -- 172.21.15.22:0/668184648 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390011840 con 0x7f03a0132150 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: dump: 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.659 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f03a57de700 5 monclient: authenticate success, global_id 4122 2023-09-30T00:44:15.660 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 1 -- 172.21.15.22:0/668184648 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03900119a0 con 0x7f03a0132150 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f03a57de700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 1 -- 172.21.15.22:0/668184648 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0390011d60 con 0x7f03a0132150 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f03a57de700 4 set_mon_vals no callback set 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.653+0000 7f039e7fc700 10 monclient: dump: 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.661 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 10 monclient: shutdown 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 -- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132150 msgr2=0x7f03a0132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 --2- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132150 0x7f03a0132790 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0390009910 tx=0x7f03900091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.662 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 -- 172.21.15.22:0/668184648 shutdown_connections 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 --2- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132150 0x7f03a0132790 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 --2- 172.21.15.22:0/668184648 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132e10 0x7f03a0098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 -- 172.21.15.22:0/668184648 >> 172.21.15.22:0/668184648 conn(0x7f03a00a54c0 msgr2=0x7f03a00a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 -- 172.21.15.22:0/668184648 shutdown_connections 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.654+0000 7f03a57de700 1 -- 172.21.15.22:0/668184648 wait complete. 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.655+0000 7f03a57de700 10 monclient: build_initial_monmap 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.655+0000 7f03a57de700 10 monclient: monmap: 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2023-09-30T00:44:15.663 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:15.664 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.664 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:15.664 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:15.664 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:15.664 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.655+0000 7f03a57de700 1 Processor -- start 2023-09-30T00:44:15.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.655+0000 7f03a57de700 1 -- start start 2023-09-30T00:44:15.668 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.655+0000 7f03a57de700 10 monclient: init 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a0098ca0 0x7f03a012e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): picked mon.noname-b con 0x7f03a0098ca0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 0x7f03a011be60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): picked mon.noname-c con 0x7f03a0132150 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132e10 0x7f03a011c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): picked mon.noname-a con 0x7f03a0132e10 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a00a1250 con 0x7f03a0132e10 2023-09-30T00:44:15.669 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03a0131380 con 0x7f03a0098ca0 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03a011fb20 con 0x7f03a0132150 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03a57de700 10 monclient(hunting): authenticate will time out at 1130.965225s 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a0098ca0 0x7f03a012e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a0098ca0 0x7f03a012e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38424/0 (socket says 172.21.15.22:38424) 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 1 -- 172.21.15.22:0/3134871409 learned_addr learned my addr 172.21.15.22:0/3134871409 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): get_auth_request con 0x7f03a0098ca0 auth_method 0 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.670 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 0x7f03a011be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132e10 0x7f03a011c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): get_auth_request con 0x7f03a0132150 auth_method 0 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): get_auth_request con 0x7f03a0132e10 auth_method 0 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:15.671 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f039f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:15.672 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient(hunting): handle_auth_done global_id 4121 payload 824 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 -- 172.21.15.22:0/3134871409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a0098ca0 msgr2=0x7f03a012e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a0098ca0 0x7f03a012e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 -- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132e10 msgr2=0x7f03a011c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132e10 0x7f03a011c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient: _finish_hunting 0 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 monclient: found mon.noname-c 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.656+0000 7f03967fc700 1 -- 172.21.15.22:0/3134871409 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0388009420 con 0x7f03a0132150 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03967fc700 1 -- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 msgr2=0x7f03a011be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:15.673 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03967fc700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 0x7f03a011be60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f038800ec50 tx=0x7f03880091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388009700 con 0x7f03a0132150 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: dump: 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.674 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f038800ca60 con 0x7f03a0132150 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 5 monclient: authenticate success, global_id 4121 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 10 monclient: _renew_subs 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038800ce00 con 0x7f03a0132150 2023-09-30T00:44:15.675 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0390009420 con 0x7f03a0132150 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f039dffb700 4 set_mon_vals no callback set 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 10 monclient: dump: 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:15.676 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 10 monclient: _renew_subs 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03a011da30 con 0x7f03a0132150 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f0388005a00 con 0x7f03a0132150 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.659+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1181+0+0 (secure 0 0 0) 0x7f0388005fd0 con 0x7f03a0132150 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.660+0000 7f03a57de700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.660+0000 7f03a57de700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.660+0000 7f03a57de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:15.677 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.660+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03a011f720 con 0x7f03a0132150 2023-09-30T00:44:15.678 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.662+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f0388021060 con 0x7f03a0132150 2023-09-30T00:44:15.678 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.662+0000 7f03977fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:15.678 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.662+0000 7f03977fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:15.751 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.749+0000 7f03a57de700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "061ce3a2-0e9b-448e-9cae-789937c85eab", "id": 4}] 2023-09-30T00:44:15.751 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.750+0000 7f03a57de700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "061ce3a2-0e9b-448e-9cae-789937c85eab", "id": 4}] 2023-09-30T00:44:15.751 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.750+0000 7f03a57de700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:15.751 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:15.750+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd new", "uuid": "061ce3a2-0e9b-448e-9cae-789937c85eab", "id": 4} v 0) v1 -- 0x7f03a0072b00 con 0x7f03a0132150 2023-09-30T00:44:16.082 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.081+0000 7f3cbdc3d1c0 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-09-30T00:44:16.162 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.161+0000 7f3cbdc3d1c0 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-09-30T00:44:16.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.202+0000 7f03977fe700 1 -- 172.21.15.22:0/3134871409 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "061ce3a2-0e9b-448e-9cae-789937c85eab", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0388017070 con 0x7f03a0132150 2023-09-30T00:44:16.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.203+0000 7f03977fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "061ce3a2-0e9b-448e-9cae-789937c85eab", "id": 4}] 2023-09-30T00:44:16.203 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.203+0000 7f03977fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:16.204 INFO:teuthology.orchestra.run.smithi022.stdout:4 2023-09-30T00:44:16.205 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.203+0000 7f03a57de700 10 monclient: shutdown 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.203+0000 7f03a57de700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 msgr2=0x7f03a011be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 0x7f03a011be60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f038800ec50 tx=0x7f03880091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 shutdown_connections 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03a0132e10 0x7f03a011c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03a0098ca0 0x7f03a012e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.206 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 --2- 172.21.15.22:0/3134871409 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03a0132150 0x7f03a011be60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.207 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 >> 172.21.15.22:0/3134871409 conn(0x7f03a00a54c0 msgr2=0x7f03a0130980 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:16.207 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 shutdown_connections 2023-09-30T00:44:16.207 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.204+0000 7f03a57de700 1 -- 172.21.15.22:0/3134871409 wait complete. 2023-09-30T00:44:16.217 DEBUG:teuthology.orchestra.run.smithi022:> sudo ceph --cluster ceph osd new 28f01a16-aeec-4d18-942d-a6a37dbbfe77 5 2023-09-30T00:44:16.316 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.316+0000 7f3cbdc3d1c0 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-09-30T00:44:16.346 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.346+0000 7f6771d4b700 10 monclient: get_monmap_and_config 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.346+0000 7f6771d4b700 10 monclient: build_initial_monmap 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.346+0000 7f6771d4b700 10 monclient: monmap: 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:44:16.347268+0000 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:44:16.347268+0000 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:16.347 INFO:teuthology.orchestra.run.smithi022.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:16.348 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:16.348 INFO:teuthology.orchestra.run.smithi022.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:16.348 INFO:teuthology.orchestra.run.smithi022.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:16.348 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 Processor -- start 2023-09-30T00:44:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 -- start start 2023-09-30T00:44:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient: init 2023-09-30T00:44:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T00:44:16.351 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f676c098670 0x7f676c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): picked mon.noname-c con 0x7f676c098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098a40 0x7f676c12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): picked mon.noname-b con 0x7f676c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f676c098e90 0x7f676c12f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): picked mon.noname-a con 0x7f676c098e90 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0a0a50 con 0x7f676c098e90 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f676c0a1000 con 0x7f676c098670 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c0a0730 con 0x7f676c098a40 2023-09-30T00:44:16.352 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.347+0000 7f6771d4b700 10 monclient(hunting): authenticate will time out at 1131.656914s 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f676c098670 0x7f676c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f676c098670 0x7f676c132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38434/0 (socket says 172.21.15.22:38434) 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 -- 172.21.15.22:0/1376140672 learned_addr learned my addr 172.21.15.22:0/1376140672 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): get_auth_request con 0x7f676c098670 auth_method 0 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 1 --2- 172.21.15.22:0/1376140672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098a40 0x7f676c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 1 --2- 172.21.15.22:0/1376140672 >> v2:172.21.15.6:3300/0 conn(0x7f676c098e90 0x7f676c12f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.353 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): get_auth_request con 0x7f676c098a40 auth_method 0 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): get_auth_request con 0x7f676c098e90 auth_method 0 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.354 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient(hunting): handle_auth_done global_id 4129 payload 274 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 -- 172.21.15.22:0/1376140672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098a40 msgr2=0x7f676c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 --2- 172.21.15.22:0/1376140672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098a40 0x7f676c12ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 -- 172.21.15.22:0/1376140672 >> v2:172.21.15.6:3300/0 conn(0x7f676c098e90 msgr2=0x7f676c12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 --2- 172.21.15.22:0/1376140672 >> v2:172.21.15.6:3300/0 conn(0x7f676c098e90 0x7f676c12f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 monclient: found mon.noname-c 2023-09-30T00:44:16.355 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.348+0000 7f676b7fe700 1 -- 172.21.15.22:0/1376140672 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c068df0 con 0x7f676c098670 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676b7fe700 1 -- 172.21.15.22:0/1376140672 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 msgr2=0x7f676c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676b7fe700 1 --2- 172.21.15.22:0/1376140672 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c132d00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f676c09fe60 tx=0x7f675c009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fdd2c3d0b92463e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676a7fc700 1 -- 172.21.15.22:0/1376140672 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00b850 con 0x7f676c098670 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676a7fc700 10 monclient: dump: 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.356 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.350+0000 7f676a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 5 monclient: authenticate success, global_id 4129 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f676a7fc700 1 -- 172.21.15.22:0/1376140672 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c00b9b0 con 0x7f676c098670 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:16.357 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f676a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f676a7fc700 1 -- 172.21.15.22:0/1376140672 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00bd50 con 0x7f676c098670 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f676a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 4 set_mon_vals no callback set 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f676a7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f676a7fc700 10 monclient: dump: 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.358 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:16.359 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.359 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.359 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.359 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:16.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 10 monclient: shutdown 2023-09-30T00:44:16.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:16.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 -- 172.21.15.22:0/1376140672 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 msgr2=0x7f676c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 --2- 172.21.15.22:0/1376140672 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c132d00 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f676c09fe60 tx=0x7f675c009250 comp rx=0 tx=0).stop 2023-09-30T00:44:16.366 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 -- 172.21.15.22:0/1376140672 shutdown_connections 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 --2- 172.21.15.22:0/1376140672 >> v2:172.21.15.6:3300/0 conn(0x7f676c098e90 0x7f676c12f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 --2- 172.21.15.22:0/1376140672 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c132d00 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 --2- 172.21.15.22:0/1376140672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098a40 0x7f676c12ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 -- 172.21.15.22:0/1376140672 >> 172.21.15.22:0/1376140672 conn(0x7f676c0a5480 msgr2=0x7f676c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 -- 172.21.15.22:0/1376140672 shutdown_connections 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.351+0000 7f6771d4b700 1 -- 172.21.15.22:0/1376140672 wait complete. 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.352+0000 7f6771d4b700 10 monclient: build_initial_monmap 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.352+0000 7f6771d4b700 10 monclient: monmap: 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 0 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 0.000000 2023-09-30T00:44:16.367 INFO:teuthology.orchestra.run.smithi022.stderr:created 0.000000 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.352+0000 7f6771d4b700 1 Processor -- start 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.352+0000 7f6771d4b700 1 -- start start 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.352+0000 7f6771d4b700 10 monclient: init 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:16.368 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c09efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): picked mon.noname-b con 0x7f676c098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676c098a40 0x7f676c141d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): picked mon.noname-a con 0x7f676c098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098e90 0x7f676c1422d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): picked mon.noname-c con 0x7f676c098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c09e8a0 con 0x7f676c098a40 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676c12fc60 con 0x7f676c098670 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f676c09e590 con 0x7f676c098e90 2023-09-30T00:44:16.369 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f6771d4b700 10 monclient(hunting): authenticate will time out at 1131.662481s 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c09efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c09efe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.22:38448/0 (socket says 172.21.15.22:38448) 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 -- 172.21.15.22:0/587927405 learned_addr learned my addr 172.21.15.22:0/587927405 (peer_addr_for_me v2:172.21.15.22:0/0) 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): get_auth_request con 0x7f676c098670 auth_method 0 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676c098a40 0x7f676c141d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.370 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098e90 0x7f676c1422d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): get_auth_request con 0x7f676c098e90 auth_method 0 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): get_auth_request con 0x7f676c098a40 auth_method 0 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.371 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient(hunting): handle_auth_done global_id 4132 payload 824 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 -- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098e90 msgr2=0x7f676c1422d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098e90 0x7f676c1422d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 -- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676c098a40 msgr2=0x7f676c141d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676c098a40 0x7f676c141d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 monclient: found mon.noname-b 2023-09-30T00:44:16.372 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.353+0000 7f676b7fe700 1 -- 172.21.15.22:0/587927405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f675c0094f0 con 0x7f676c098670 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f676b7fe700 1 -- 172.21.15.22:0/587927405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 msgr2=0x7f676c09efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f676b7fe700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c09efe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f675c006fd0 tx=0x7f675c017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c011bd0 con 0x7f676c098670 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: dump: 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.373 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f675c011d70 con 0x7f676c098670 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 5 monclient: authenticate success, global_id 4132 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 10 monclient: _renew_subs 2023-09-30T00:44:16.374 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f676c09dfd0 con 0x7f676c098670 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c024690 con 0x7f676c098670 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 10 monclient: dump: 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:epoch 1 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.375 INFO:teuthology.orchestra.run.smithi022.stderr:election_strategy: 1 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr: 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6769ffb700 4 set_mon_vals no callback set 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 10 monclient: _renew_subs 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f676c09f520 con 0x7f676c098670 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f675c024d40 con 0x7f676c098670 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.356+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1296+0+0 (secure 0 0 0) 0x7f675c02de20 con 0x7f676c098670 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.357+0000 7f6771d4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:16.376 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.357+0000 7f6771d4b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.357+0000 7f6771d4b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.357+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676c0980d0 con 0x7f676c098670 2023-09-30T00:44:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.359+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f675c015020 con 0x7f676c098670 2023-09-30T00:44:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.359+0000 7f67537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:16.377 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.359+0000 7f67537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:16.395 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.395+0000 7f3cbdc3d1c0 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-09-30T00:44:16.448 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.447+0000 7f6771d4b700 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "28f01a16-aeec-4d18-942d-a6a37dbbfe77", "id": 5}] 2023-09-30T00:44:16.448 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.447+0000 7f6771d4b700 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "28f01a16-aeec-4d18-942d-a6a37dbbfe77", "id": 5}] 2023-09-30T00:44:16.448 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.447+0000 7f6771d4b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:16.448 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.447+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd new", "uuid": "28f01a16-aeec-4d18-942d-a6a37dbbfe77", "id": 5} v 0) v1 -- 0x7f676c072b00 con 0x7f676c098670 2023-09-30T00:44:16.451 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.451+0000 7f67537fe700 1 -- 172.21.15.22:0/587927405 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "28f01a16-aeec-4d18-942d-a6a37dbbfe77", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f675c02d440 con 0x7f676c098670 2023-09-30T00:44:16.451 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.451+0000 7f67537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "28f01a16-aeec-4d18-942d-a6a37dbbfe77", "id": 5}] 2023-09-30T00:44:16.451 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.451+0000 7f67537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:16.451 INFO:teuthology.orchestra.run.smithi022.stdout:5 2023-09-30T00:44:16.452 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 10 monclient: shutdown 2023-09-30T00:44:16.452 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:16.452 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 msgr2=0x7f676c09efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.452 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c09efe0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f675c006fd0 tx=0x7f675c017000 comp rx=0 tx=0).stop 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 shutdown_connections 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f676c098e90 0x7f676c1422d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f676c098670 0x7f676c09efe0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 --2- 172.21.15.22:0/587927405 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f676c098a40 0x7f676c141d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 >> 172.21.15.22:0/587927405 conn(0x7f676c0a5480 msgr2=0x7f676c133b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 shutdown_connections 2023-09-30T00:44:16.453 INFO:teuthology.orchestra.run.smithi022.stderr:2023-09-30T00:44:16.452+0000 7f6771d4b700 1 -- 172.21.15.22:0/587927405 wait complete. 2023-09-30T00:44:16.465 INFO:tasks.ceph.osd.0:Restarting daemon 2023-09-30T00:44:16.466 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2023-09-30T00:44:16.468 INFO:tasks.ceph.osd.0:Started 2023-09-30T00:44:16.468 INFO:tasks.ceph.osd.1:Restarting daemon 2023-09-30T00:44:16.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2023-09-30T00:44:16.470 INFO:tasks.ceph.osd.1:Started 2023-09-30T00:44:16.471 INFO:tasks.ceph.osd.2:Restarting daemon 2023-09-30T00:44:16.471 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2023-09-30T00:44:16.473 INFO:tasks.ceph.osd.2:Started 2023-09-30T00:44:16.473 INFO:tasks.ceph.osd.3:Restarting daemon 2023-09-30T00:44:16.473 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2023-09-30T00:44:16.475 INFO:tasks.ceph.osd.3:Started 2023-09-30T00:44:16.475 INFO:tasks.ceph.osd.4:Restarting daemon 2023-09-30T00:44:16.475 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2023-09-30T00:44:16.477 INFO:tasks.ceph.osd.4:Started 2023-09-30T00:44:16.478 INFO:tasks.ceph.osd.5:Restarting daemon 2023-09-30T00:44:16.478 DEBUG:teuthology.orchestra.run.smithi022:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2023-09-30T00:44:16.480 INFO:tasks.ceph.osd.5:Started 2023-09-30T00:44:16.480 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:16.612 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.611+0000 7f3cbdc3d1c0 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.629+0000 7f45faadc700 10 monclient: get_monmap_and_config 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.629+0000 7f45faadc700 10 monclient: build_initial_monmap 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.629+0000 7f45faadc700 10 monclient: monmap: 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:16.630715+0000 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:16.630715+0000 2023-09-30T00:44:16.630 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:16.631 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 Processor -- start 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 -- start start 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient: init 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 0x7f45f4132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): picked mon.noname-b con 0x7f45f4132150 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f45f4132e10 0x7f45f4098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): picked mon.noname-c con 0x7f45f4132e10 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45f4098ca0 0x7f45f4068df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): picked mon.noname-a con 0x7f45f4098ca0 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f4099120 con 0x7f45f4098ca0 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45f4132520 con 0x7f45f4132e10 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4096a80 con 0x7f45f4132150 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.630+0000 7f45faadc700 10 monclient(hunting): authenticate will time out at 1252.928918s 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45f4098ca0 0x7f45f4068df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 0x7f45f4132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 0x7f45f4132790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42976/0 (socket says 172.21.15.6:42976) 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f45f4098ca0 0x7f45f4068df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49612/0 (socket says 172.21.15.6:49612) 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 1 -- 172.21.15.6:0/2437662108 learned_addr learned my addr 172.21.15.6:0/2437662108 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): get_auth_request con 0x7f45f4132150 auth_method 0 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): get_auth_request con 0x7f45f4098ca0 auth_method 0 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 1 --2- 172.21.15.6:0/2437662108 >> v2:172.21.15.22:3300/0 conn(0x7f45f4132e10 0x7f45f4098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): get_auth_request con 0x7f45f4132e10 auth_method 0 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient(hunting): handle_auth_done global_id 4143 payload 274 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 -- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 msgr2=0x7f45f4132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 --2- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 0x7f45f4132790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 -- 172.21.15.6:0/2437662108 >> v2:172.21.15.22:3300/0 conn(0x7f45f4132e10 msgr2=0x7f45f4098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 --2- 172.21.15.6:0/2437662108 >> v2:172.21.15.22:3300/0 conn(0x7f45f4132e10 0x7f45f4098760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient: _finish_hunting 0 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 monclient: found mon.noname-a 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:16.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f9079700 1 -- 172.21.15.6:0/2437662108 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f40a0f80 con 0x7f45f4098ca0 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f3fff700 1 --2- 172.21.15.6:0/2437662108 >> v2:172.21.15.22:3300/0 conn(0x7f45f4132e10 0x7f45f4098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.631+0000 7f45f8878700 1 --2- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 0x7f45f4132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.633+0000 7f45f9079700 1 -- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4098ca0 msgr2=0x7f45f4068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.633+0000 7f45f9079700 1 --2- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4098ca0 0x7f45f4068df0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009e60 tx=0x7f45ec009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ace32090336d7d0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.633+0000 7f45f37fe700 1 -- 172.21.15.6:0/2437662108 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec00e070 con 0x7f45f4098ca0 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.633+0000 7f45f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: dump: 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 1 -- 172.21.15.6:0/2437662108 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45ec00bc80 con 0x7f45f4098ca0 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 5 monclient: authenticate success, global_id 4143 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 1 -- 172.21.15.6:0/2437662108 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45ec012430 con 0x7f45f4098ca0 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 4 set_mon_vals no callback set 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45f37fe700 10 monclient: dump: 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 10 monclient: shutdown 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 -- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4098ca0 msgr2=0x7f45f4068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 --2- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4098ca0 0x7f45f4068df0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f45ec009e60 tx=0x7f45ec009780 comp rx=0 tx=0).stop 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 -- 172.21.15.6:0/2437662108 shutdown_connections 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 --2- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4098ca0 0x7f45f4068df0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 --2- 172.21.15.6:0/2437662108 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132150 0x7f45f4132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 --2- 172.21.15.6:0/2437662108 >> v2:172.21.15.22:3300/0 conn(0x7f45f4132e10 0x7f45f4098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 -- 172.21.15.6:0/2437662108 >> 172.21.15.6:0/2437662108 conn(0x7f45f40a54c0 msgr2=0x7f45f40a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 -- 172.21.15.6:0/2437662108 shutdown_connections 2023-09-30T00:44:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.634+0000 7f45faadc700 1 -- 172.21.15.6:0/2437662108 wait complete. 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.635+0000 7f45faadc700 10 monclient: build_initial_monmap 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.635+0000 7f45faadc700 10 monclient: monmap: 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.635+0000 7f45faadc700 1 Processor -- start 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.635+0000 7f45faadc700 1 -- start start 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.635+0000 7f45faadc700 10 monclient: init 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45f4098ca0 0x7f45f412e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): picked mon.noname-b con 0x7f45f4098ca0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 0x7f45f412ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): picked mon.noname-a con 0x7f45f4132150 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 0x7f45f409c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): picked mon.noname-c con 0x7f45f4132e10 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f411c4a0 con 0x7f45f4132150 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f409e970 con 0x7f45f4098ca0 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45f411c1c0 con 0x7f45f4132e10 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45faadc700 10 monclient(hunting): authenticate will time out at 1252.934270s 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 0x7f45f409c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 0x7f45f412ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 0x7f45f412ef30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49628/0 (socket says 172.21.15.6:49628) 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 0x7f45f409c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42988/0 (socket says 172.21.15.6:42988) 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 -- 172.21.15.6:0/2590520993 learned_addr learned my addr 172.21.15.6:0/2590520993 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): get_auth_request con 0x7f45f4132150 auth_method 0 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): get_auth_request con 0x7f45f4132e10 auth_method 0 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45f4098ca0 0x7f45f412e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): get_auth_request con 0x7f45f4098ca0 auth_method 0 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient(hunting): handle_auth_done global_id 4146 payload 824 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 -- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 msgr2=0x7f45f409c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 0x7f45f409c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 -- 172.21.15.6:0/2590520993 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45f4098ca0 msgr2=0x7f45f412e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45f4098ca0 0x7f45f412e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 monclient: found mon.noname-a 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f3fff700 1 -- 172.21.15.6:0/2590520993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45e40094f0 con 0x7f45f4132150 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f8878700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45f4098ca0 0x7f45f412e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.636+0000 7f45f9079700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 0x7f45f409c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f3fff700 1 -- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 msgr2=0x7f45f412ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f3fff700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 0x7f45f412ef30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00ec80 tx=0x7f45dc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc00ca60 con 0x7f45f4132150 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: dump: 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 5 monclient: authenticate success, global_id 4146 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45dc00cbc0 con 0x7f45f4132150 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 10 monclient: _renew_subs 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45dc009420 con 0x7f45f4132150 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45dc013470 con 0x7f45f4132150 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 10 monclient: dump: 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 10 monclient: _renew_subs 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f2ffd700 4 set_mon_vals no callback set 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45ec009a20 con 0x7f45f4132150 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f45dc013a60 con 0x7f45f4132150 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.639+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f45dc0129d0 con 0x7f45f4132150 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.640+0000 7f45faadc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.640+0000 7f45faadc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.640+0000 7f45faadc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.640+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f4098710 con 0x7f45f4132150 2023-09-30T00:44:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.642+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f45dc021060 con 0x7f45f4132150 2023-09-30T00:44:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.642+0000 7f45f0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.642+0000 7f45f0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:16.698 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.698+0000 7f3cbdc3d1c0 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-09-30T00:44:16.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.714+0000 7f45faadc700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:16.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.714+0000 7f45faadc700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:16.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.714+0000 7f45faadc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:16.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.714+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45f409e4b0 con 0x7f45f4132150 2023-09-30T00:44:16.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.715+0000 7f45f0ff9700 1 -- 172.21.15.6:0/2590520993 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f45dc017070 con 0x7f45f4132150 2023-09-30T00:44:16.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.715+0000 7f45f0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:16.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.715+0000 7f45f0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:16.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:16.717 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":8,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:16.449558+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:16.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 10 monclient: shutdown 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 msgr2=0x7f45f412ef30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 0x7f45f412ef30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f45dc00ec80 tx=0x7f45dc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 shutdown_connections 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f45f4132e10 0x7f45f409c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f45f4098ca0 0x7f45f412e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 --2- 172.21.15.6:0/2590520993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f45f4132150 0x7f45f412ef30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 >> 172.21.15.6:0/2590520993 conn(0x7f45f40a54c0 msgr2=0x7f45f4097500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 shutdown_connections 2023-09-30T00:44:16.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:16.716+0000 7f45faadc700 1 -- 172.21.15.6:0/2590520993 wait complete. 2023-09-30T00:44:16.730 INFO:tasks.ceph.ceph_manager.ceph:[] 2023-09-30T00:44:16.730 INFO:tasks.ceph:Waiting for OSDs to come up 2023-09-30T00:44:16.784 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.783+0000 7f3cbdc3d1c0 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-09-30T00:44:16.838 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-09-30T00:44:16.837+0000 7f69c7bc7540 -1 Falling back to public interface 2023-09-30T00:44:16.880 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:16.880+0000 7f3cbdc3d1c0 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-09-30T00:44:17.079 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:17.079+0000 7f3cbdc3d1c0 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-09-30T00:44:17.083 INFO:tasks.ceph.osd.1.smithi006.stderr:2023-09-30T00:44:17.083+0000 7fbd4331b540 -1 Falling back to public interface 2023-09-30T00:44:17.084 INFO:tasks.ceph.osd.0.smithi006.stderr:2023-09-30T00:44:17.084+0000 7fa5b311a540 -1 Falling back to public interface 2023-09-30T00:44:17.087 INFO:tasks.ceph.osd.2.smithi006.stderr:2023-09-30T00:44:17.087+0000 7f4f61f79540 -1 Falling back to public interface 2023-09-30T00:44:17.093 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-09-30T00:44:17.093+0000 7fa569fd8540 -1 Falling back to public interface 2023-09-30T00:44:17.094 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-09-30T00:44:17.093+0000 7efcdc87f540 -1 Falling back to public interface 2023-09-30T00:44:17.170 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:17.169+0000 7f3cbdc3d1c0 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-09-30T00:44:17.333 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.463+0000 7f1253af2700 10 monclient: get_monmap_and_config 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.463+0000 7f1253af2700 10 monclient: build_initial_monmap 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.463+0000 7f1253af2700 10 monclient: monmap: 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:44:17.464391+0000 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:44:17.464391+0000 2023-09-30T00:44:17.464 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:17.465 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:17.467 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.463+0000 7f1253af2700 1 Processor -- start 2023-09-30T00:44:17.467 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 1 -- start start 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient: init 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 0x7f124c132850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): picked mon.noname-b con 0x7f124c098650 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f124c098a20 0x7f124c09fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): picked mon.noname-c con 0x7f124c098a20 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f124c098e50 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f124c06a6e0 con 0x7f124c098e50 2023-09-30T00:44:17.468 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f124c096e40 con 0x7f124c098a20 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124c06a430 con 0x7f124c098650 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f1253af2700 10 monclient(hunting): authenticate will time out at 1253.762584s 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 0x7f124c132850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 0x7f124c132850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:43004/0 (socket says 172.21.15.6:43004) 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 1 -- 172.21.15.6:0/95312837 learned_addr learned my addr 172.21.15.6:0/95312837 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): get_auth_request con 0x7f124c098650 auth_method 0 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125208f700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f124c098e50 0x7f124c0a03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57636/0 (socket says 172.21.15.6:57636) 2023-09-30T00:44:17.469 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 1 --2- 172.21.15.6:0/95312837 >> v2:172.21.15.22:3300/0 conn(0x7f124c098a20 0x7f124c09fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): get_auth_request con 0x7f124c098a20 auth_method 0 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:17.470 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125108d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.464+0000 7f125188e700 10 monclient(hunting): handle_auth_done global_id 4145 payload 274 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 1 -- 172.21.15.6:0/95312837 >> v2:172.21.15.22:3300/0 conn(0x7f124c098a20 msgr2=0x7f124c09fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 1 --2- 172.21.15.6:0/95312837 >> v2:172.21.15.22:3300/0 conn(0x7f124c098a20 0x7f124c09fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 1 -- 172.21.15.6:0/95312837 >> v1:172.21.15.6:6789/0 conn(0x7f124c098e50 legacy=0x7f124c0a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 10 monclient: _finish_hunting 0 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 1 monclient: found mon.noname-b 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.465+0000 7f125188e700 1 -- 172.21.15.6:0/95312837 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c096b60 con 0x7f124c098650 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125188e700 1 -- 172.21.15.6:0/95312837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 msgr2=0x7f124c132850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:17.471 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125188e700 1 --2- 172.21.15.6:0/95312837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 0x7f124c132850 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f12480099b0 tx=0x7f1248009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3963ab11975a25b9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 1 -- 172.21.15.6:0/95312837 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1248009ea0 con 0x7f124c098650 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: dump: 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:17.477 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: _finish_auth 0 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 1 -- 172.21.15.6:0/95312837 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f124800ba30 con 0x7f124c098650 2023-09-30T00:44:17.478 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f1253af2700 5 monclient: authenticate success, global_id 4145 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f1253af2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 1 -- 172.21.15.6:0/95312837 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124800bdf0 con 0x7f124c098650 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f125088c700 10 monclient: dump: 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:17.479 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f1253af2700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.467+0000 7f1253af2700 4 set_mon_vals no callback set 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 10 monclient: shutdown 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 -- 172.21.15.6:0/95312837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 msgr2=0x7f124c132850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:17.480 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 --2- 172.21.15.6:0/95312837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 0x7f124c132850 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f12480099b0 tx=0x7f1248009250 comp rx=0 tx=0).stop 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 -- 172.21.15.6:0/95312837 shutdown_connections 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 --2- 172.21.15.6:0/95312837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098650 0x7f124c132850 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 --2- 172.21.15.6:0/95312837 >> v2:172.21.15.22:3300/0 conn(0x7f124c098a20 0x7f124c09fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 -- 172.21.15.6:0/95312837 >> 172.21.15.6:0/95312837 conn(0x7f124c0a1fa0 msgr2=0x7f124c0a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 -- 172.21.15.6:0/95312837 shutdown_connections 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 1 -- 172.21.15.6:0/95312837 wait complete. 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 10 monclient: build_initial_monmap 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.468+0000 7f1253af2700 10 monclient: monmap: 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2023-09-30T00:44:17.481 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 Processor -- start 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 -- start start 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient: init 2023-09-30T00:44:17.482 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 0x7f124c11c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): picked mon.noname-a con 0x7f124c098650 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 0x7f124c11c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): picked mon.noname-b con 0x7f124c098a20 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098e50 0x7f124c11cc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): picked mon.noname-c con 0x7f124c098e50 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c132500 con 0x7f124c098650 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:17.483 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c0a51a0 con 0x7f124c098a20 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f124c11eb50 con 0x7f124c098e50 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f1253af2700 10 monclient(hunting): authenticate will time out at 1253.767625s 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 0x7f124c11c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098e50 0x7f124c11cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 0x7f124c11c1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49638/0 (socket says 172.21.15.6:49638) 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 1 -- 172.21.15.6:0/2432414092 learned_addr learned my addr 172.21.15.6:0/2432414092 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:17.484 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): get_auth_request con 0x7f124c098650 auth_method 0 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): get_auth_request con 0x7f124c098e50 auth_method 0 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 0x7f124c11c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:17.485 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): get_auth_request con 0x7f124c098a20 auth_method 0 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125188e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125208f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:17.486 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.469+0000 7f125108d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 10 monclient(hunting): handle_auth_done global_id 4153 payload 824 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 1 -- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098e50 msgr2=0x7f124c11cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098e50 0x7f124c11cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 1 -- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 msgr2=0x7f124c11c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 0x7f124c11c1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 10 monclient: _finish_hunting 0 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 1 monclient: found mon.noname-b 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125108d700 1 -- 172.21.15.6:0/2432414092 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12480094f0 con 0x7f124c098a20 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125188e700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 0x7f124c11c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:17.487 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.470+0000 7f125208f700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098e50 0x7f124c11cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f125108d700 1 -- 172.21.15.6:0/2432414092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 msgr2=0x7f124c11c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f125108d700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 0x7f124c11c710 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f123c00ea50 tx=0x7f123c0091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c0095d0 con 0x7f124c098a20 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: dump: 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.488 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f123c00ca00 con 0x7f124c098a20 2023-09-30T00:44:17.489 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 5 monclient: authenticate success, global_id 4153 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 10 monclient: _renew_subs 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c11f930 con 0x7f124c098a20 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f123c00cdc0 con 0x7f124c098a20 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1241ffb700 10 monclient: dump: 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:17.490 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1243fff700 4 set_mon_vals no callback set 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 10 monclient: _renew_subs 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.472+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c12f9c0 con 0x7f124c098a20 2023-09-30T00:44:17.491 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.473+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f123c005c80 con 0x7f124c098a20 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.473+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1411+0+0 (secure 0 0 0) 0x7f123c013c20 con 0x7f124c098a20 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.473+0000 7f1253af2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.473+0000 7f1253af2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.473+0000 7f1253af2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.473+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c0a00e0 con 0x7f124c098a20 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.476+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f123c021060 con 0x7f124c098a20 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.476+0000 7f1241ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:17.492 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.476+0000 7f1241ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:17.495 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:17.495+0000 7f3cbdc3d1c0 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-09-30T00:44:17.549 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.548+0000 7f1253af2700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:17.549 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.548+0000 7f1253af2700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:17.550 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.548+0000 7f1253af2700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:17.550 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.548+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f124c0649b0 con 0x7f124c098a20 2023-09-30T00:44:17.551 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1241ffb700 1 -- 172.21.15.6:0/2432414092 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) v1 ==== 74+0+5560 (secure 0 0 0) 0x7f123c017070 con 0x7f124c098a20 2023-09-30T00:44:17.551 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1241ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:17.551 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1241ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:17.552 INFO:teuthology.misc.health.smithi006.stdout: 2023-09-30T00:44:17.552 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":8,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:16.449558+0000","last_up_change":"0.000000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:17.554 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 10 monclient: shutdown 2023-09-30T00:44:17.554 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:17.554 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 msgr2=0x7f124c11c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:17.555 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 0x7f124c11c710 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f123c00ea50 tx=0x7f123c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.555 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 shutdown_connections 2023-09-30T00:44:17.555 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f124c098e50 0x7f124c11cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.555 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f124c098650 0x7f124c11c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.555 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 --2- 172.21.15.6:0/2432414092 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f124c098a20 0x7f124c11c710 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:17.556 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 >> 172.21.15.6:0/2432414092 conn(0x7f124c0a1fa0 msgr2=0x7f124c09b390 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:17.556 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 shutdown_connections 2023-09-30T00:44:17.556 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:17.550+0000 7f1253af2700 1 -- 172.21.15.6:0/2432414092 wait complete. 2023-09-30T00:44:17.562 DEBUG:teuthology.misc:0 of 6 OSDs are up 2023-09-30T00:44:17.574 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:17.574+0000 7f3cbdc3d1c0 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-09-30T00:44:18.148 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:18.147+0000 7f3cbdc3d1c0 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-09-30T00:44:18.507 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:18.507+0000 7f3cbdc3d1c0 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-09-30T00:44:18.664 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:18.664+0000 7f3cbdc3d1c0 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:44:19.067 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.066+0000 7f3cbdc3d1c0 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-09-30T00:44:19.229 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.227+0000 7f3cbdc3d1c0 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-09-30T00:44:19.402 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.402+0000 7f3cbdc3d1c0 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-09-30T00:44:19.515 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.515+0000 7f3cbdc3d1c0 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-09-30T00:44:19.571 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.571+0000 7f3c4d0e7700 -1 log_channel(cluster) log [ERR] : Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:44:19.572 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.571+0000 7f3c4d0e7700 -1 pg_autoscaler.serve: 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:19.571+0000 7f3c4d0e7700 -1 Traceback (most recent call last): 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 325, in serve 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr: self._maybe_adjust() 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr: File "/usr/share/ceph/mgr/pg_autoscaler/module.py", line 682, in _maybe_adjust 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr: if self.noautoscale: 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr:AttributeError: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:44:19.573 INFO:tasks.ceph.mgr.x.smithi022.stderr: 2023-09-30T00:44:20.344 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-09-30T00:44:20.343+0000 7f69c7bc7540 -1 osd.3 0 log_to_monitors true 2023-09-30T00:44:20.554 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:20.552+0000 7f3c82321700 -1 mgr.server handle_report got status from non-daemon mon.c 2023-09-30T00:44:20.554 INFO:tasks.ceph.mgr.x.smithi022.stderr:2023-09-30T00:44:20.552+0000 7f3c82321700 -1 mgr.server handle_report got status from non-daemon mon.b 2023-09-30T00:44:20.554 INFO:tasks.ceph.osd.1.smithi006.stderr:2023-09-30T00:44:20.553+0000 7fbd4331b540 -1 osd.1 0 log_to_monitors true 2023-09-30T00:44:20.592 INFO:tasks.ceph.osd.0.smithi006.stderr:2023-09-30T00:44:20.591+0000 7fa5b311a540 -1 osd.0 0 log_to_monitors true 2023-09-30T00:44:20.606 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-09-30T00:44:20.604+0000 7fa569fd8540 -1 osd.5 0 log_to_monitors true 2023-09-30T00:44:20.609 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-09-30T00:44:20.608+0000 7efcdc87f540 -1 osd.4 0 log_to_monitors true 2023-09-30T00:44:20.615 INFO:tasks.ceph.osd.2.smithi006.stderr:2023-09-30T00:44:20.615+0000 7f4f61f79540 -1 osd.2 0 log_to_monitors true 2023-09-30T00:44:22.223 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-09-30T00:44:22.222+0000 7f69bffcf700 -1 osd.3 0 waiting for initial osdmap 2023-09-30T00:44:22.224 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-09-30T00:44:22.222+0000 7fa560bdd700 -1 osd.5 0 waiting for initial osdmap 2023-09-30T00:44:22.224 INFO:tasks.ceph.osd.2.smithi006.stderr:2023-09-30T00:44:22.223+0000 7f4f58b7e700 -1 osd.2 0 waiting for initial osdmap 2023-09-30T00:44:22.227 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-09-30T00:44:22.226+0000 7efcd3484700 -1 osd.4 0 waiting for initial osdmap 2023-09-30T00:44:22.228 INFO:tasks.ceph.osd.1.smithi006.stderr:2023-09-30T00:44:22.228+0000 7fbd39f20700 -1 osd.1 0 waiting for initial osdmap 2023-09-30T00:44:22.229 INFO:tasks.ceph.osd.0.smithi006.stderr:2023-09-30T00:44:22.228+0000 7fa5a9d1f700 -1 osd.0 0 waiting for initial osdmap 2023-09-30T00:44:22.232 INFO:tasks.ceph.osd.3.smithi022.stderr:2023-09-30T00:44:22.231+0000 7f69b9dc8700 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:44:22.233 INFO:tasks.ceph.osd.5.smithi022.stderr:2023-09-30T00:44:22.231+0000 7fa55c9da700 -1 osd.5 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:44:22.233 INFO:tasks.ceph.osd.4.smithi022.stderr:2023-09-30T00:44:22.232+0000 7efccf281700 -1 osd.4 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:44:22.234 INFO:tasks.ceph.osd.1.smithi006.stderr:2023-09-30T00:44:22.233+0000 7fbd35d1d700 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:44:22.234 INFO:tasks.ceph.osd.2.smithi006.stderr:2023-09-30T00:44:22.233+0000 7f4f5497b700 -1 osd.2 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:44:22.235 INFO:tasks.ceph.osd.0.smithi006.stderr:2023-09-30T00:44:22.234+0000 7fa5a5b1c700 -1 osd.0 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-09-30T00:44:24.167 DEBUG:teuthology.orchestra.run.smithi006:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:24.231 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:44:24.230+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : Health check failed: Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' (MGR_MODULE_ERROR) 2023-09-30T00:44:24.301 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.300+0000 7f4b4de49700 10 monclient: get_monmap_and_config 2023-09-30T00:44:24.301 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.300+0000 7f4b4de49700 10 monclient: build_initial_monmap 2023-09-30T00:44:24.301 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.300+0000 7f4b4de49700 10 monclient: monmap: 2023-09-30T00:44:24.301 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2023-09-30T00:44:24.301 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:24.301 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:44:24.301525+0000 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:44:24.301525+0000 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:24.302 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 Processor -- start 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 -- start start 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient: init 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4b48098650 0x7f4b48132850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): picked mon.noname-c con 0x7f4b48098650 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4809fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:24.306 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): picked mon.noname-b con 0x7f4b48098a20 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4b48098e50 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4806a6e0 con 0x7f4b48098e50 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b48096e40 con 0x7f4b48098650 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4806a430 con 0x7f4b48098a20 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b4de49700 10 monclient(hunting): authenticate will time out at 1260.599715s 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4809fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b47fff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4b48098e50 0x7f4b480a03e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57646/0 (socket says 172.21.15.6:57646) 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b47fff700 1 -- 172.21.15.6:0/2202150592 learned_addr learned my addr 172.21.15.6:0/2202150592 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:24.307 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4809fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:43056/0 (socket says 172.21.15.6:43056) 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b46ffd700 10 monclient(hunting): get_auth_request con 0x7f4b48098a20 auth_method 0 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b46ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b46ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.301+0000 7f4b477fe700 1 --2- 172.21.15.6:0/2202150592 >> v2:172.21.15.22:3300/0 conn(0x7f4b48098650 0x7f4b48132850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): get_auth_request con 0x7f4b48098650 auth_method 0 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:24.308 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_done global_id 4187 payload 274 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2202150592 >> v2:172.21.15.22:3300/0 conn(0x7f4b48098650 msgr2=0x7f4b48132850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 1 --2- 172.21.15.6:0/2202150592 >> v2:172.21.15.22:3300/0 conn(0x7f4b48098650 0x7f4b48132850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2202150592 >> v1:172.21.15.6:6789/0 conn(0x7f4b48098e50 legacy=0x7f4b480a03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 1 monclient: found mon.noname-b 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:24.309 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.302+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2202150592 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b48096b60 con 0x7f4b48098a20 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2202150592 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 msgr2=0x7f4b4809fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b46ffd700 1 --2- 172.21.15.6:0/2202150592 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4809fea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0098e0 tx=0x7f4b3c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d44536f3c72428c2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 1 -- 172.21.15.6:0/2202150592 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c011840 con 0x7f4b48098a20 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: dump: 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.310 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 1 -- 172.21.15.6:0/2202150592 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c0119a0 con 0x7f4b48098a20 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b4de49700 5 monclient: authenticate success, global_id 4187 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b4de49700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:24.311 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 1 -- 172.21.15.6:0/2202150592 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c011d40 con 0x7f4b48098a20 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b4de49700 4 set_mon_vals no callback set 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.304+0000 7f4b467fc700 10 monclient: dump: 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:24.312 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 10 monclient: shutdown 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 -- 172.21.15.6:0/2202150592 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 msgr2=0x7f4b4809fea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2202150592 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4809fea0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0098e0 tx=0x7f4b3c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 -- 172.21.15.6:0/2202150592 shutdown_connections 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2202150592 >> v2:172.21.15.22:3300/0 conn(0x7f4b48098650 0x7f4b48132850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2202150592 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4809fea0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 -- 172.21.15.6:0/2202150592 >> 172.21.15.6:0/2202150592 conn(0x7f4b480a1fa0 msgr2=0x7f4b480a23a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:24.313 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 -- 172.21.15.6:0/2202150592 shutdown_connections 2023-09-30T00:44:24.314 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 1 -- 172.21.15.6:0/2202150592 wait complete. 2023-09-30T00:44:24.318 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 10 monclient: build_initial_monmap 2023-09-30T00:44:24.318 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.305+0000 7f4b4de49700 10 monclient: monmap: 2023-09-30T00:44:24.318 INFO:teuthology.misc.health.smithi006.stderr:epoch 0 2023-09-30T00:44:24.318 INFO:teuthology.misc.health.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:24.318 INFO:teuthology.misc.health.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:24.318 INFO:teuthology.misc.health.smithi006.stderr:created 0.000000 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 Processor -- start 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 -- start start 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient: init 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:24.319 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b48098650 0x7f4b4811c190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): picked mon.noname-a con 0x7f4b48098650 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): picked mon.noname-c con 0x7f4b48098a20 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b48098e50 0x7f4b4811cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): picked mon.noname-b con 0x7f4b48098e50 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b48132500 con 0x7f4b48098650 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b480a5160 con 0x7f4b48098e50 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b4811fbd0 con 0x7f4b48098a20 2023-09-30T00:44:24.320 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.306+0000 7f4b4de49700 10 monclient(hunting): authenticate will time out at 1260.604974s 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:43070/0 (socket says 172.21.15.6:43070) 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b48098650 0x7f4b4811c190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2220691247 learned_addr learned my addr 172.21.15.6:0/2220691247 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): get_auth_request con 0x7f4b48098a20 auth_method 0 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): get_auth_request con 0x7f4b48098650 auth_method 0 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:24.321 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b48098e50 0x7f4b4811cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): get_auth_request con 0x7f4b48098e50 auth_method 0 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:24.322 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b47fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient(hunting): handle_auth_done global_id 4190 payload 824 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2220691247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b48098e50 msgr2=0x7f4b4811cc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b48098e50 0x7f4b4811cc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b48098650 msgr2=0x7f4b4811c190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b48098650 0x7f4b4811c190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 monclient: found mon.noname-c 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:24.323 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2220691247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b3c009420 con 0x7f4b48098a20 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.307+0000 7f4b477fe700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b48098650 0x7f4b4811c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b46ffd700 1 -- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 msgr2=0x7f4b4811c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b46ffd700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0093f0 tx=0x7f4b3c016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c006af0 con 0x7f4b48098a20 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: dump: 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.324 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c020070 con 0x7f4b48098a20 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b4de49700 5 monclient: authenticate success, global_id 4190 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4b3c00bd20 con 0x7f4b48098a20 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b4de49700 10 monclient: _renew_subs 2023-09-30T00:44:24.325 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b4de49700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b4812f8f0 con 0x7f4b48098a20 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b2f7fe700 10 monclient: dump: 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:epoch 1 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:24.326 INFO:teuthology.misc.health.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr: 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.309+0000 7f4b45ffb700 4 set_mon_vals no callback set 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 10 monclient: _renew_subs 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b4811f910 con 0x7f4b48098a20 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b46ffd700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0093f0 tx=0x7f4b3c016000 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f4b3c0492a0 osd_map(12..12 src has 1..12) v4 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 4) v1 ==== 42450+0+0 (secure 0 0 0) 0x7f4b3c022030 con 0x7f4b48098a20 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:24.327 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.310+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b4811ff60 con 0x7f4b48098a20 2023-09-30T00:44:24.328 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.311+0000 7f4b2f7fe700 1 --2- 172.21.15.6:0/2220691247 >> v2:172.21.15.22:6812/142603 conn(0x7f4b34035710 0x7f4b34037bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:24.328 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.311+0000 7f4b477fe700 1 --2- 172.21.15.6:0/2220691247 >> v2:172.21.15.22:6812/142603 conn(0x7f4b34035710 0x7f4b34037bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:24.328 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.311+0000 7f4b477fe700 10 monclient: get_auth_request con 0x7f4b34035710 auth_method 0 2023-09-30T00:44:24.328 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.314+0000 7f4b477fe700 1 --2- 172.21.15.6:0/2220691247 >> v2:172.21.15.22:6812/142603 conn(0x7f4b34035710 0x7f4b34037bc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4b38011480 tx=0x7f4b380091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:24.328 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:24.316+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(12..12 src has 1..12) v4 ==== 3138+0+0 (secure 0 0 0) 0x7f4b3c0492a0 con 0x7f4b48098a20 2023-09-30T00:44:25.138 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.139+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4b4811ff60 con 0x7f4b48098a20 2023-09-30T00:44:25.139 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.139+0000 7f4b2f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:25.139 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.139+0000 7f4b2f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:25.348 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.347+0000 7f4b4de49700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:25.348 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.347+0000 7f4b4de49700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:25.349 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.347+0000 7f4b4de49700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:25.349 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.347+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4b4811ff60 con 0x7f4b48098a20 2023-09-30T00:44:25.349 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.348+0000 7f4b2f7fe700 1 -- 172.21.15.6:0/2220691247 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v13) v1 ==== 74+0+8470 (secure 0 0 0) 0x7f4b4811ff60 con 0x7f4b48098a20 2023-09-30T00:44:25.349 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.348+0000 7f4b2f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:25.349 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.348+0000 7f4b2f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:25.350 INFO:teuthology.misc.health.smithi006.stdout: 2023-09-30T00:44:25.350 INFO:teuthology.misc.health.smithi006.stdout:{"epoch":13,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:25.239467+0000","last_up_change":"2023-09-30T00:44:23.223970+0000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:44:23.604669+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":114301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":114301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":114301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":114301}]},"public_addr":"172.21.15.6:6804/114301","cluster_addr":"172.21.15.6:6805/114301","heartbeat_back_addr":"172.21.15.6:6807/114301","heartbeat_front_addr":"172.21.15.6:6806/114301","state":["exists","up"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":114300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":114300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":114300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":114300}]},"public_addr":"172.21.15.6:6800/114300","cluster_addr":"172.21.15.6:6801/114300","heartbeat_back_addr":"172.21.15.6:6803/114300","heartbeat_front_addr":"172.21.15.6:6802/114300","state":["exists","up"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":114302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":114302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":114302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":114302}]},"public_addr":"172.21.15.6:6808/114302","cluster_addr":"172.21.15.6:6809/114302","heartbeat_back_addr":"172.21.15.6:6811/114302","heartbeat_front_addr":"172.21.15.6:6810/114302","state":["exists","up"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":143135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6801","nonce":143135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6803","nonce":143135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":143135}]},"public_addr":"172.21.15.22:6800/143135","cluster_addr":"172.21.15.22:6801/143135","heartbeat_back_addr":"172.21.15.22:6803/143135","heartbeat_front_addr":"172.21.15.22:6802/143135","state":["exists","up"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":143134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":143134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6811","nonce":143134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":143134}]},"public_addr":"172.21.15.22:6808/143134","cluster_addr":"172.21.15.22:6809/143134","heartbeat_back_addr":"172.21.15.22:6811/143134","heartbeat_front_addr":"172.21.15.22:6810/143134","state":["exists","up"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":143133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6805","nonce":143133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6807","nonce":143133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":143133}]},"public_addr":"172.21.15.22:6804/143133","cluster_addr":"172.21.15.22:6805/143133","heartbeat_back_addr":"172.21.15.22:6807/143133","heartbeat_front_addr":"172.21.15.22:6806/143133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.597432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.528045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.614505+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.340159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.641614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.578369+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 >> v2:172.21.15.22:6812/142603 conn(0x7f4b34035710 msgr2=0x7f4b34037bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2220691247 >> v2:172.21.15.22:6812/142603 conn(0x7f4b34035710 0x7f4b34037bc0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4b38011480 tx=0x7f4b380091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 10 monclient: shutdown 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 msgr2=0x7f4b4811c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4b3c0093f0 tx=0x7f4b3c016000 comp rx=0 tx=0).stop 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.350+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 shutdown_connections 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2220691247 >> v2:172.21.15.22:6812/142603 conn(0x7f4b34035710 0x7f4b34037bc0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4b48098e50 0x7f4b4811cc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.352 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4b48098650 0x7f4b4811c190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.353 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 --2- 172.21.15.6:0/2220691247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4b48098a20 0x7f4b4811c6d0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.353 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 >> 172.21.15.6:0/2220691247 conn(0x7f4b480a1fa0 msgr2=0x7f4b481319e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:25.353 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 shutdown_connections 2023-09-30T00:44:25.353 INFO:teuthology.misc.health.smithi006.stderr:2023-09-30T00:44:25.351+0000 7f4b4de49700 1 -- 172.21.15.6:0/2220691247 wait complete. 2023-09-30T00:44:25.363 DEBUG:teuthology.misc:6 of 6 OSDs are up 2023-09-30T00:44:25.364 INFO:tasks.ceph:Creating RBD pool 2023-09-30T00:44:25.364 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool create rbd 8 2023-09-30T00:44:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.493+0000 7f455542a700 10 monclient: get_monmap_and_config 2023-09-30T00:44:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.493+0000 7f455542a700 10 monclient: build_initial_monmap 2023-09-30T00:44:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.493+0000 7f455542a700 10 monclient: monmap: 2023-09-30T00:44:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:25.494 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:25.494399+0000 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:25.494399+0000 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.493+0000 7f455542a700 1 Processor -- start 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.493+0000 7f455542a700 1 -- start start 2023-09-30T00:44:25.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.493+0000 7f455542a700 10 monclient: init 2023-09-30T00:44:25.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4550098670 0x7f4550132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): picked mon.noname-a con 0x7f4550098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): picked mon.noname-b con 0x7f4550098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4550098e90 0x7f455012f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): picked mon.noname-c con 0x7f4550098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45500a0a50 con 0x7f4550098670 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:25.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45500a1000 con 0x7f4550098e90 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45500a0730 con 0x7f4550098a40 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f455542a700 10 monclient(hunting): authenticate will time out at 1261.792416s 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4550098670 0x7f4550132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4550098670 0x7f4550132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49668/0 (socket says 172.21.15.6:49668) 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:43080/0 (socket says 172.21.15.6:43080) 2023-09-30T00:44:25.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 -- 172.21.15.6:0/1024580877 learned_addr learned my addr 172.21.15.6:0/1024580877 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): get_auth_request con 0x7f4550098670 auth_method 0 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): get_auth_request con 0x7f4550098a40 auth_method 0 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 1 --2- 172.21.15.6:0/1024580877 >> v2:172.21.15.22:3300/0 conn(0x7f4550098e90 0x7f455012f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): get_auth_request con 0x7f4550098e90 auth_method 0 2023-09-30T00:44:25.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient(hunting): handle_auth_done global_id 4188 payload 274 2023-09-30T00:44:25.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 -- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 msgr2=0x7f455012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 --2- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 -- 172.21.15.6:0/1024580877 >> v2:172.21.15.22:3300/0 conn(0x7f4550098e90 msgr2=0x7f455012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 --2- 172.21.15.6:0/1024580877 >> v2:172.21.15.22:3300/0 conn(0x7f4550098e90 0x7f455012f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 monclient: found mon.noname-a 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454effd700 1 -- 172.21.15.6:0/1024580877 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4550068df0 con 0x7f4550098670 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.494+0000 7f454e7fc700 1 --2- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454effd700 1 -- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098670 msgr2=0x7f4550132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:25.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454effd700 1 --2- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098670 0x7f4550132d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f45500a1df0 tx=0x7f4540009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10a7a4c599c70453 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 1 -- 172.21.15.6:0/1024580877 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454000b850 con 0x7f4550098670 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: dump: 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:25.505 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 1 -- 172.21.15.6:0/1024580877 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454000b9b0 con 0x7f4550098670 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 5 monclient: authenticate success, global_id 4188 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 1 -- 172.21.15.6:0/1024580877 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454000bd50 con 0x7f4550098670 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 4 set_mon_vals no callback set 2023-09-30T00:44:25.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f454dffb700 10 monclient: dump: 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:25.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 10 monclient: shutdown 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 1 -- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098670 msgr2=0x7f4550132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.497+0000 7f455542a700 1 --2- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098670 0x7f4550132d00 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f45500a1df0 tx=0x7f4540009250 comp rx=0 tx=0).stop 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 -- 172.21.15.6:0/1024580877 shutdown_connections 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 --2- 172.21.15.6:0/1024580877 >> v2:172.21.15.22:3300/0 conn(0x7f4550098e90 0x7f455012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 --2- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098670 0x7f4550132d00 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 --2- 172.21.15.6:0/1024580877 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 -- 172.21.15.6:0/1024580877 >> 172.21.15.6:0/1024580877 conn(0x7f45500a5480 msgr2=0x7f45500a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 -- 172.21.15.6:0/1024580877 shutdown_connections 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 1 -- 172.21.15.6:0/1024580877 wait complete. 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 10 monclient: build_initial_monmap 2023-09-30T00:44:25.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.498+0000 7f455542a700 10 monclient: monmap: 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 Processor -- start 2023-09-30T00:44:25.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 -- start start 2023-09-30T00:44:25.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient: init 2023-09-30T00:44:25.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:25.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:25.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4550098670 0x7f455012d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): picked mon.noname-b con 0x7f4550098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:25.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): picked mon.noname-c con 0x7f4550098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098e90 0x7f455009d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): picked mon.noname-a con 0x7f4550098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550135390 con 0x7f4550098e90 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f455009d220 con 0x7f4550098670 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f455012ed20 con 0x7f4550098a40 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f455542a700 10 monclient(hunting): authenticate will time out at 1261.797867s 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f454e7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f454e7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:43088/0 (socket says 172.21.15.6:43088) 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f454f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098e90 0x7f455009d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f454e7fc700 1 -- 172.21.15.6:0/840133094 learned_addr learned my addr 172.21.15.6:0/840133094 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f454e7fc700 10 monclient(hunting): get_auth_request con 0x7f4550098a40 auth_method 0 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.499+0000 7f454e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): get_auth_request con 0x7f4550098e90 auth_method 0 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4550098670 0x7f455012d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): get_auth_request con 0x7f4550098670 auth_method 0 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:25.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient(hunting): handle_auth_done global_id 4196 payload 824 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 1 -- 172.21.15.6:0/840133094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4550098670 msgr2=0x7f455012d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4550098670 0x7f455012d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 1 -- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098e90 msgr2=0x7f455009d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098e90 0x7f455009d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 1 monclient: found mon.noname-c 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.500+0000 7f454e7fc700 1 -- 172.21.15.6:0/840133094 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544009420 con 0x7f4550098a40 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f454e7fc700 1 -- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 msgr2=0x7f455012d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f454e7fc700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f454400ec80 tx=0x7f45440091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:25.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454400ca60 con 0x7f4550098a40 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: dump: 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:25.519 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: _finish_auth 0 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 5 monclient: authenticate success, global_id 4196 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f454400cbc0 con 0x7f4550098a40 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4544013470 con 0x7f4550098a40 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f4536ffd700 10 monclient: dump: 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:25.520 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 10 monclient: _renew_subs 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f454d7fa700 4 set_mon_vals no callback set 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45400094f0 con 0x7f4550098a40 2023-09-30T00:44:25.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 10 monclient: _renew_subs 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.502+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f455012f3e0 con 0x7f4550098a40 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f454e7fc700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f454400ec80 tx=0x7f45440091a0 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f45440137e0 mgrmap(e 4) v1 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 4) v1 ==== 42450+0+0 (secure 0 0 0) 0x7f45440137e0 con 0x7f4550098a40 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f455542a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f455542a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f455542a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f4536ffd700 1 --2- 172.21.15.6:0/840133094 >> v2:172.21.15.22:6812/142603 conn(0x7f453c035710 0x7f453c037bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.503+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f455009efe0 con 0x7f4550098a40 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.504+0000 7f454effd700 1 --2- 172.21.15.6:0/840133094 >> v2:172.21.15.22:6812/142603 conn(0x7f453c035710 0x7f453c037bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:25.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.504+0000 7f454effd700 10 monclient: get_auth_request con 0x7f453c035710 auth_method 0 2023-09-30T00:44:25.523 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.506+0000 7f454effd700 1 --2- 172.21.15.6:0/840133094 >> v2:172.21.15.22:6812/142603 conn(0x7f453c035710 0x7f453c037bc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f45500a1df0 tx=0x7f4540017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:25.523 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:25.508+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 3149+0+0 (secure 0 0 0) 0x7f4544047bb0 con 0x7f4550098a40 2023-09-30T00:44:26.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.332+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f45440238e0 con 0x7f4550098a40 2023-09-30T00:44:26.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.332+0000 7f4536ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:26.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.332+0000 7f4536ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:26.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.545+0000 7f455542a700 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T00:44:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.545+0000 7f455542a700 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T00:44:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.545+0000 7f455542a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:26.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:26.545+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7f4550072b00 con 0x7f4550098a40 2023-09-30T00:44:27.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.408+0000 7f4536ffd700 1 -- 172.21.15.6:0/840133094 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v15) v1 ==== 109+0+0 (secure 0 0 0) 0x7f45440232f0 con 0x7f4550098a40 2023-09-30T00:44:27.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.408+0000 7f4536ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2023-09-30T00:44:27.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.408+0000 7f4536ffd700 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2023-09-30T00:44:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 >> v2:172.21.15.22:6812/142603 conn(0x7f453c035710 msgr2=0x7f453c037bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 --2- 172.21.15.6:0/840133094 >> v2:172.21.15.22:6812/142603 conn(0x7f453c035710 0x7f453c037bc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f45500a1df0 tx=0x7f4540017000 comp rx=0 tx=0).stop 2023-09-30T00:44:27.411 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 10 monclient: shutdown 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 msgr2=0x7f455012d8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f454400ec80 tx=0x7f45440091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 shutdown_connections 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 --2- 172.21.15.6:0/840133094 >> v2:172.21.15.22:6812/142603 conn(0x7f453c035710 0x7f453c037bc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4550098e90 0x7f455009d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4550098670 0x7f455012d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 --2- 172.21.15.6:0/840133094 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4550098a40 0x7f455012d8a0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 >> 172.21.15.6:0/840133094 conn(0x7f45500a5480 msgr2=0x7f45500a3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 shutdown_connections 2023-09-30T00:44:27.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.411+0000 7f455542a700 1 -- 172.21.15.6:0/840133094 wait complete. 2023-09-30T00:44:27.413 INFO:teuthology.orchestra.run.smithi006.stderr:pool 'rbd' created 2023-09-30T00:44:27.429 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2023-09-30T00:44:27.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.566+0000 7f2cc488a700 10 monclient: get_monmap_and_config 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.566+0000 7f2cc488a700 10 monclient: build_initial_monmap 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.566+0000 7f2cc488a700 10 monclient: monmap: 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:27.566357+0000 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:27.566357+0000 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:27.566 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:27.567 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:27.567 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.566+0000 7f2cc488a700 1 Processor -- start 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.566+0000 7f2cc488a700 1 -- start start 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.566+0000 7f2cc488a700 10 monclient: init 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2cc0098530 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:27.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cc0132360 0x7f2cc0132760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): picked mon.noname-a con 0x7f2cc0132360 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 0x7f2cc0139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): picked mon.noname-b con 0x7f2cc01374a0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2cc0096af0 con 0x7f2cc0098530 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2cc006a420 con 0x7f2cc0132360 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0097000 con 0x7f2cc01374a0 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cc488a700 10 monclient(hunting): authenticate will time out at 1263.864627s 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 0x7f2cc0139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:27.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cc0132360 0x7f2cc0132760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 0x7f2cc0139880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36380/0 (socket says 172.21.15.6:36380) 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2cc0132360 0x7f2cc0132760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41844/0 (socket says 172.21.15.6:41844) 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 1 -- 172.21.15.6:0/348257672 learned_addr learned my addr 172.21.15.6:0/348257672 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): get_auth_request con 0x7f2cc01374a0 auth_method 0 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): get_auth_request con 0x7f2cc0132360 auth_method 0 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:27.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.567+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_done global_id 4203 payload 274 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 1 -- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 msgr2=0x7f2cc0139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 1 --2- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 0x7f2cc0139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 1 -- 172.21.15.6:0/348257672 >> v1:172.21.15.22:6789/0 conn(0x7f2cc0098530 legacy=0x7f2cc0098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 1 monclient: found mon.noname-a 2023-09-30T00:44:27.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbdd9b700 1 -- 172.21.15.6:0/348257672 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cc0132ca0 con 0x7f2cc0132360 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.568+0000 7f2cbed9d700 1 --2- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 0x7f2cc0139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbdd9b700 1 -- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 msgr2=0x7f2cc0132760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbdd9b700 1 --2- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc0132760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80098e0 tx=0x7f2ca80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1fc9a2835e9cb52 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 1 -- 172.21.15.6:0/348257672 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8011840 con 0x7f2cc0132360 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: dump: 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:27.575 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: _finish_auth 0 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cc488a700 5 monclient: authenticate success, global_id 4203 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cc488a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:27.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 1 -- 172.21.15.6:0/348257672 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ca80119a0 con 0x7f2cc0132360 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cc488a700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 1 -- 172.21.15.6:0/348257672 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ca8011d40 con 0x7f2cc0132360 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cc488a700 4 set_mon_vals no callback set 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.570+0000 7f2cbd59a700 10 monclient: dump: 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:27.577 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 10 monclient: shutdown 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 -- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 msgr2=0x7f2cc0132760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 --2- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc0132760 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80098e0 tx=0x7f2ca80091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 -- 172.21.15.6:0/348257672 shutdown_connections 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 --2- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc01374a0 0x7f2cc0139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 --2- 172.21.15.6:0/348257672 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc0132760 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 -- 172.21.15.6:0/348257672 >> 172.21.15.6:0/348257672 conn(0x7f2cc00a5460 msgr2=0x7f2cc00a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 -- 172.21.15.6:0/348257672 shutdown_connections 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 1 -- 172.21.15.6:0/348257672 wait complete. 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 10 monclient: build_initial_monmap 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.571+0000 7f2cc488a700 10 monclient: monmap: 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:27.579 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:27.580 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:27.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 Processor -- start 2023-09-30T00:44:27.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 -- start start 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient: init 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 0x7f2cc0141c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): picked mon.noname-c con 0x7f2cc0098530 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc01421d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): picked mon.noname-a con 0x7f2cc0132360 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2cc01374a0 0x7f2cc0142710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): picked mon.noname-b con 0x7f2cc01374a0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:27.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0135fc0 con 0x7f2cc0132360 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0134a80 con 0x7f2cc01374a0 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2cc0135970 con 0x7f2cc0098530 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cc488a700 10 monclient(hunting): authenticate will time out at 1263.869713s 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 0x7f2cc0141c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 0x7f2cc0141c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36396/0 (socket says 172.21.15.6:36396) 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 1 -- 172.21.15.6:0/4065313843 learned_addr learned my addr 172.21.15.6:0/4065313843 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): get_auth_request con 0x7f2cc0098530 auth_method 0 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:27.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbdd9b700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc01421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbed9d700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2cc01374a0 0x7f2cc0142710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.572+0000 7f2cbe59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): get_auth_request con 0x7f2cc0132360 auth_method 0 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): get_auth_request con 0x7f2cc01374a0 auth_method 0 2023-09-30T00:44:27.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 10 monclient(hunting): handle_auth_done global_id 4211 payload 824 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 1 -- 172.21.15.6:0/4065313843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2cc01374a0 msgr2=0x7f2cc0142710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2cc01374a0 0x7f2cc0142710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 1 -- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 msgr2=0x7f2cc01421d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:27.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc01421d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 10 monclient: _finish_hunting 0 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 1 monclient: found mon.noname-c 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.573+0000 7f2cbe59c700 1 -- 172.21.15.6:0/4065313843 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cb8009ae0 con 0x7f2cc0098530 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cbe59c700 1 -- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 msgr2=0x7f2cc0141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cbe59c700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 0x7f2cc0141c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000c5d0 tx=0x7f2cb0007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb00117a0 con 0x7f2cc0098530 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: dump: 2023-09-30T00:44:27.589 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:27.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cb0011900 con 0x7f2cc0098530 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 5 monclient: authenticate success, global_id 4211 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 10 monclient: _renew_subs 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca8009420 con 0x7f2cc0098530 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0011cc0 con 0x7f2cc0098530 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cb67fc700 10 monclient: dump: 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:27.591 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cbcd99700 4 set_mon_vals no callback set 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 10 monclient: _renew_subs 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.575+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cc009f630 con 0x7f2cc0098530 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.576+0000 7f2cc488a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:27.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.576+0000 7f2cc488a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.576+0000 7f2cc488a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.576+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cc01362a0 con 0x7f2cc0098530 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.576+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 4) v1 ==== 42450+0+0 (secure 0 0 0) 0x7f2cb0012630 con 0x7f2cc0098530 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.577+0000 7f2cb67fc700 1 --2- 172.21.15.6:0/4065313843 >> v2:172.21.15.22:6812/142603 conn(0x7f2ca0035710 0x7f2ca0037bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.577+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 3491+0+0 (secure 0 0 0) 0x7f2cb0047570 con 0x7f2cc0098530 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.577+0000 7f2cbdd9b700 1 --2- 172.21.15.6:0/4065313843 >> v2:172.21.15.22:6812/142603 conn(0x7f2ca0035710 0x7f2ca0037bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.577+0000 7f2cbdd9b700 10 monclient: get_auth_request con 0x7f2ca0035710 auth_method 0 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.579+0000 7f2cbdd9b700 1 --2- 172.21.15.6:0/4065313843 >> v2:172.21.15.22:6812/142603 conn(0x7f2ca0035710 0x7f2ca0037bc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80030c0 tx=0x7f2ca8016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.582+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2cb00108d0 con 0x7f2cc0098530 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.582+0000 7f2cb67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:27.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.582+0000 7f2cb67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:27.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.785+0000 7f2cc488a700 10 start_mon_command cmd=[{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T00:44:27.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.785+0000 7f2cc488a700 10 monclient: _send_command 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T00:44:27.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.785+0000 7f2cc488a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:27.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:27.785+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f2cc0135e10 con 0x7f2cc0098530 2023-09-30T00:44:28.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.419+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2cb001c7e0 con 0x7f2cc0098530 2023-09-30T00:44:28.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.419+0000 7f2cb67fc700 1 -- 172.21.15.6:0/4065313843 <== mon.2 v2:172.21.15.6:3301/0 8 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v16) v1 ==== 173+0+0 (secure 0 0 0) 0x7f2cb0049500 con 0x7f2cc0098530 2023-09-30T00:44:28.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.419+0000 7f2cb67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}] 2023-09-30T00:44:28.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.419+0000 7f2cb67fc700 10 monclient: _finish_command 2 = system:0 enabled application 'rbd' on pool 'rbd' 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 >> v2:172.21.15.22:6812/142603 conn(0x7f2ca0035710 msgr2=0x7f2ca0037bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 --2- 172.21.15.6:0/4065313843 >> v2:172.21.15.22:6812/142603 conn(0x7f2ca0035710 0x7f2ca0037bc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f2ca80030c0 tx=0x7f2ca8016000 comp rx=0 tx=0).stop 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 10 monclient: shutdown 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 msgr2=0x7f2cc0141c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 0x7f2cc0141c90 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f2cb000c5d0 tx=0x7f2cb0007f60 comp rx=0 tx=0).stop 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 shutdown_connections 2023-09-30T00:44:28.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 --2- 172.21.15.6:0/4065313843 >> v2:172.21.15.22:6812/142603 conn(0x7f2ca0035710 0x7f2ca0037bc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2cc01374a0 0x7f2cc0142710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2cc0098530 0x7f2cc0141c90 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 --2- 172.21.15.6:0/4065313843 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2cc0132360 0x7f2cc01421d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.423+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 >> 172.21.15.6:0/4065313843 conn(0x7f2cc00a5460 msgr2=0x7f2cc012ee60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.424+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 shutdown_connections 2023-09-30T00:44:28.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.424+0000 7f2cc488a700 1 -- 172.21.15.6:0/4065313843 wait complete. 2023-09-30T00:44:28.425 INFO:teuthology.orchestra.run.smithi006.stderr:enabled application 'rbd' on pool 'rbd' 2023-09-30T00:44:28.439 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2023-09-30T00:44:28.439 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-09-30T00:44:28.439 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2023-09-30T00:44:28.440 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2023-09-30T00:44:28.440 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2023-09-30T00:44:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.574+0000 7f03c205d700 10 monclient: get_monmap_and_config 2023-09-30T00:44:28.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.574+0000 7f03c205d700 10 monclient: build_initial_monmap 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.574+0000 7f03c205d700 10 monclient: monmap: 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:28.574781+0000 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:28.574781+0000 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:28.575 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:28.576 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:28.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 1 Processor -- start 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 1 -- start start 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 10 monclient: init 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03bc136a80 0x7f03bc136e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 10 monclient(hunting): picked mon.noname-a con 0x7f03bc136a80 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f03bc136600 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.575+0000 7f03c205d700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f03bc098a90 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03c205d700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc097e50 con 0x7f03bc136600 2023-09-30T00:44:28.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03c205d700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03bc098130 con 0x7f03bc098a90 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03c205d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03c205d700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03bc0a0c10 con 0x7f03bc136a80 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03c205d700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03c205d700 10 monclient(hunting): authenticate will time out at 1264.873048s 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f03bc136a80 0x7f03bc136e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03baffd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f03bc136600 0x7f03bc09fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60818/0 (socket says 172.21.15.6:60818) 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03baffd700 1 -- 172.21.15.6:0/400382600 learned_addr learned my addr 172.21.15.6:0/400382600 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): get_auth_request con 0x7f03bc136a80 auth_method 0 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:28.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_done global_id 4209 payload 274 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 1 -- 172.21.15.6:0/400382600 >> v1:172.21.15.22:6789/0 conn(0x7f03bc098a90 legacy=0x7f03bc0a0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 1 -- 172.21.15.6:0/400382600 >> v1:172.21.15.6:6789/0 conn(0x7f03bc136600 legacy=0x7f03bc09fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 1 monclient: found mon.noname-a 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:28.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.576+0000 7f03bb7fe700 1 -- 172.21.15.6:0/400382600 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03bc097b70 con 0x7f03bc136a80 2023-09-30T00:44:28.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03bb7fe700 1 -- 172.21.15.6:0/400382600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136a80 msgr2=0x7f03bc136e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03bb7fe700 1 --2- 172.21.15.6:0/400382600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136a80 0x7f03bc136e50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009e60 tx=0x7f03ac009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c610d7eddf64279 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 1 -- 172.21.15.6:0/400382600 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac00e070 con 0x7f03bc136a80 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 10 monclient: dump: 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:28.586 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03ba7fc700 1 -- 172.21.15.6:0/400382600 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03ac012430 con 0x7f03bc136a80 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03c205d700 5 monclient: authenticate success, global_id 4209 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.578+0000 7f03c205d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03ba7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:28.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03ba7fc700 1 -- 172.21.15.6:0/400382600 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03ac0127f0 con 0x7f03bc136a80 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 4 set_mon_vals no callback set 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03ba7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03ba7fc700 10 monclient: dump: 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:28.588 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 10 monclient: shutdown 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 -- 172.21.15.6:0/400382600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136a80 msgr2=0x7f03bc136e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 --2- 172.21.15.6:0/400382600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136a80 0x7f03bc136e50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009e60 tx=0x7f03ac009780 comp rx=0 tx=0).stop 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 -- 172.21.15.6:0/400382600 shutdown_connections 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 --2- 172.21.15.6:0/400382600 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136a80 0x7f03bc136e50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 -- 172.21.15.6:0/400382600 >> 172.21.15.6:0/400382600 conn(0x7f03bc0a1ef0 msgr2=0x7f03bc0a22f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 -- 172.21.15.6:0/400382600 shutdown_connections 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.579+0000 7f03c205d700 1 -- 172.21.15.6:0/400382600 wait complete. 2023-09-30T00:44:28.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.580+0000 7f03c205d700 10 monclient: build_initial_monmap 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.580+0000 7f03c205d700 10 monclient: monmap: 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:28.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.580+0000 7f03c205d700 1 Processor -- start 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.580+0000 7f03c205d700 1 -- start start 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.580+0000 7f03c205d700 10 monclient: init 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03bc098a90 0x7f03bc06de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): picked mon.noname-c con 0x7f03bc098a90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 0x7f03bc06e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): picked mon.noname-a con 0x7f03bc136600 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03bc136a80 0x7f03bc06e880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): picked mon.noname-b con 0x7f03bc136a80 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:28.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc124e20 con 0x7f03bc136600 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc135480 con 0x7f03bc136a80 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03bc126ba0 con 0x7f03bc098a90 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03c205d700 10 monclient(hunting): authenticate will time out at 1264.878273s 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 0x7f03bc06e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03bc098a90 0x7f03bc06de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 0x7f03bc06e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41876/0 (socket says 172.21.15.6:41876) 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 -- 172.21.15.6:0/2968314628 learned_addr learned my addr 172.21.15.6:0/2968314628 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): get_auth_request con 0x7f03bc136600 auth_method 0 2023-09-30T00:44:28.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): get_auth_request con 0x7f03bc098a90 auth_method 0 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:28.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bbfff700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03bc136a80 0x7f03bc06e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bbfff700 10 monclient(hunting): get_auth_request con 0x7f03bc136a80 auth_method 0 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03bbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient(hunting): handle_auth_done global_id 4212 payload 824 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 -- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03bc098a90 msgr2=0x7f03bc06de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03bc098a90 0x7f03bc06de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 -- 172.21.15.6:0/2968314628 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03bc136a80 msgr2=0x7f03bc06e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03bc136a80 0x7f03bc06e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:28.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 monclient: found mon.noname-a 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.581+0000 7f03baffd700 1 -- 172.21.15.6:0/2968314628 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03ac009a20 con 0x7f03bc136600 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.583+0000 7f03baffd700 1 -- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 msgr2=0x7f03bc06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.583+0000 7f03baffd700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 0x7f03bc06e340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c5e0 tx=0x7f03a4004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4005d40 con 0x7f03bc136600 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: dump: 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:28.595 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 5 monclient: authenticate success, global_id 4212 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a40070d0 con 0x7f03bc136600 2023-09-30T00:44:28.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 10 monclient: _renew_subs 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03bc096ee0 con 0x7f03bc136600 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03a4007490 con 0x7f03bc136600 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b9ffb700 4 set_mon_vals no callback set 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03b37fe700 10 monclient: dump: 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:28.597 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 10 monclient: _renew_subs 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.584+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03bc1243b0 con 0x7f03bc136600 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f03a401b3b0 con 0x7f03bc136600 2023-09-30T00:44:28.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03c205d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03c205d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03c205d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03b37fe700 1 --2- 172.21.15.6:0/2968314628 >> v2:172.21.15.22:6812/142603 conn(0x7f039c0357a0 0x7f039c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03bc0a4170 con 0x7f03bc136600 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f03a4046eb0 con 0x7f03bc136600 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03bb7fe700 1 --2- 172.21.15.6:0/2968314628 >> v2:172.21.15.22:6812/142603 conn(0x7f039c0357a0 0x7f039c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:28.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.585+0000 7f03bb7fe700 10 monclient: get_auth_request con 0x7f039c0357a0 auth_method 0 2023-09-30T00:44:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.588+0000 7f03bb7fe700 1 --2- 172.21.15.6:0/2968314628 >> v2:172.21.15.22:6812/142603 conn(0x7f039c0357a0 0x7f039c037c50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0097c0 tx=0x7f03ac018000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.590+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f03a401be60 con 0x7f03bc136600 2023-09-30T00:44:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.590+0000 7f03b37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:28.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.590+0000 7f03b37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.849+0000 7f03c205d700 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2023-09-30T00:44:28.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.849+0000 7f03c205d700 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T00:44:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.849+0000 7f03c205d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:28.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.849+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f03bc072b00 con 0x7f03bc136600 2023-09-30T00:44:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.851+0000 7f03b37fe700 1 -- 172.21.15.6:0/2968314628 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) v1 ==== 74+0+82384 (secure 0 0 0) 0x7f03a4017030 con 0x7f03bc136600 2023-09-30T00:44:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.851+0000 7f03b37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2023-09-30T00:44:28.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.851+0000 7f03b37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:28.852 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 >> v2:172.21.15.22:6812/142603 conn(0x7f039c0357a0 msgr2=0x7f039c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 --2- 172.21.15.6:0/2968314628 >> v2:172.21.15.22:6812/142603 conn(0x7f039c0357a0 0x7f039c037c50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f03ac0097c0 tx=0x7f03ac018000 comp rx=0 tx=0).stop 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 10 monclient: shutdown 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 msgr2=0x7f03bc06e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 0x7f03bc06e340 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f03a400c5e0 tx=0x7f03a4004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 shutdown_connections 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 --2- 172.21.15.6:0/2968314628 >> v2:172.21.15.22:6812/142603 conn(0x7f039c0357a0 0x7f039c037c50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f03bc136a80 0x7f03bc06e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f03bc098a90 0x7f03bc06de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 --2- 172.21.15.6:0/2968314628 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f03bc136600 0x7f03bc06e340 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 >> 172.21.15.6:0/2968314628 conn(0x7f03bc0a1ef0 msgr2=0x7f03bc09b880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 shutdown_connections 2023-09-30T00:44:28.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:28.854+0000 7f03c205d700 1 -- 172.21.15.6:0/2968314628 wait complete. 2023-09-30T00:44:28.867 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":5,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6812","nonce":142603}]},"active_addr":"172.21.15.22:6812/142603","active_change":"2023-09-30T00:44:19.529414+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":3444215211}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":2277256020}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":3313713050}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.22:0","nonce":3691633110}]}]} 2023-09-30T00:44:28.870 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2023-09-30T00:44:28.870 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2023-09-30T00:44:28.870 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.000+0000 7fa5c3b24700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.000+0000 7fa5c3b24700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.000+0000 7fa5c3b24700 10 monclient: monmap: 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.000445+0000 2023-09-30T00:44:29.000 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.000445+0000 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.000+0000 7fa5c3b24700 1 Processor -- start 2023-09-30T00:44:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 1 -- start start 2023-09-30T00:44:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient: init 2023-09-30T00:44:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T00:44:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5bc098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 0x7fa5bc137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 0x7fa5bc139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): picked mon.noname-a con 0x7fa5bc137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5bc0a1000 con 0x7fa5bc098670 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa5bc099150 con 0x7fa5bc137b60 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc0a0a50 con 0x7fa5bc132d00 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c3b24700 10 monclient(hunting): authenticate will time out at 1265.298663s 2023-09-30T00:44:29.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 0x7fa5bc137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 0x7fa5bc139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 0x7fa5bc137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36422/0 (socket says 172.21.15.6:36422) 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 0x7fa5bc139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41882/0 (socket says 172.21.15.6:41882) 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 1 -- 172.21.15.6:0/1621077969 learned_addr learned my addr 172.21.15.6:0/1621077969 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): get_auth_request con 0x7fa5bc132d00 auth_method 0 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): get_auth_request con 0x7fa5bc137b60 auth_method 0 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.001+0000 7fa5c20c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_done global_id 4217 payload 274 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 1 -- 172.21.15.6:0/1621077969 >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 msgr2=0x7fa5bc139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 1 --2- 172.21.15.6:0/1621077969 >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 0x7fa5bc139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 1 -- 172.21.15.6:0/1621077969 >> v1:172.21.15.6:6789/0 conn(0x7fa5bc098670 legacy=0x7fa5bc098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 1 monclient: found mon.noname-b 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c10bf700 1 -- 172.21.15.6:0/1621077969 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5bc0a0730 con 0x7fa5bc132d00 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.002+0000 7fa5c20c1700 1 --2- 172.21.15.6:0/1621077969 >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 0x7fa5bc139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c10bf700 1 -- 172.21.15.6:0/1621077969 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 msgr2=0x7fa5bc137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c10bf700 1 --2- 172.21.15.6:0/1621077969 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 0x7fa5bc137620 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0099b0 tx=0x7fa5ac009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbd770334ab43b61 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 1 -- 172.21.15.6:0/1621077969 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac009ea0 con 0x7fa5bc132d00 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: dump: 2023-09-30T00:44:29.011 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 1 -- 172.21.15.6:0/1621077969 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5ac00ba30 con 0x7fa5bc132d00 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c3b24700 5 monclient: authenticate success, global_id 4217 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c3b24700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c3b24700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 1 -- 172.21.15.6:0/1621077969 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac00bdf0 con 0x7fa5bc132d00 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c3b24700 4 set_mon_vals no callback set 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.004+0000 7fa5c08be700 10 monclient: dump: 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 10 monclient: shutdown 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1621077969 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 msgr2=0x7fa5bc137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1621077969 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 0x7fa5bc137620 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac0099b0 tx=0x7fa5ac009250 comp rx=0 tx=0).stop 2023-09-30T00:44:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1621077969 shutdown_connections 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1621077969 >> v2:172.21.15.6:3300/0 conn(0x7fa5bc137b60 0x7fa5bc139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1621077969 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc132d00 0x7fa5bc137620 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1621077969 >> 172.21.15.6:0/1621077969 conn(0x7fa5bc0a5480 msgr2=0x7fa5bc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1621077969 shutdown_connections 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1621077969 wait complete. 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.005+0000 7fa5c3b24700 10 monclient: monmap: 2023-09-30T00:44:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.017 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 Processor -- start 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 -- start start 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient: init 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 0x7fa5bc12f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): picked mon.noname-c con 0x7fa5bc098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 0x7fa5bc09c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): picked mon.noname-a con 0x7fa5bc132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5bc137b60 0x7fa5bc09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): picked mon.noname-b con 0x7fa5bc137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc1364e0 con 0x7fa5bc132d00 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc133780 con 0x7fa5bc137b60 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5bc137780 con 0x7fa5bc098670 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c3b24700 10 monclient(hunting): authenticate will time out at 1265.303756s 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 0x7fa5bc12f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 0x7fa5bc09c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 0x7fa5bc12f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36438/0 (socket says 172.21.15.6:36438) 2023-09-30T00:44:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 0x7fa5bc09c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41886/0 (socket says 172.21.15.6:41886) 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 1 -- 172.21.15.6:0/1627484541 learned_addr learned my addr 172.21.15.6:0/1627484541 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 10 monclient(hunting): get_auth_request con 0x7fa5bc098670 auth_method 0 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c18c0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): get_auth_request con 0x7fa5bc132d00 auth_method 0 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.006+0000 7fa5c10bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5bc137b60 0x7fa5bc09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): get_auth_request con 0x7fa5bc137b60 auth_method 0 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c20c1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 10 monclient(hunting): handle_auth_done global_id 4220 payload 824 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 1 -- 172.21.15.6:0/1627484541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5bc137b60 msgr2=0x7fa5bc09cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5bc137b60 0x7fa5bc09cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 1 -- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 msgr2=0x7fa5bc09c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 0x7fa5bc09c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c18c0700 1 -- 172.21.15.6:0/1627484541 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b4009a20 con 0x7fa5bc098670 2023-09-30T00:44:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.007+0000 7fa5c10bf700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 0x7fa5bc09c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5c18c0700 1 -- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 msgr2=0x7fa5bc12f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5c18c0700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 0x7fa5bc12f3a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c5d0 tx=0x7fa5b8007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b80117a0 con 0x7fa5bc098670 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: dump: 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5c3b24700 5 monclient: authenticate success, global_id 4220 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5b8011900 con 0x7fa5bc098670 2023-09-30T00:44:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5c3b24700 10 monclient: _renew_subs 2023-09-30T00:44:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8011cc0 con 0x7fa5bc098670 2023-09-30T00:44:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5c3b24700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5ac0094f0 con 0x7fa5bc098670 2023-09-30T00:44:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b3fff700 4 set_mon_vals no callback set 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.009+0000 7fa5b1ffb700 10 monclient: dump: 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.010+0000 7fa5c3b24700 10 monclient: _renew_subs 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.010+0000 7fa5c3b24700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.010+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5bc09e580 con 0x7fa5bc098670 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.010+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa5b8012630 con 0x7fa5bc098670 2023-09-30T00:44:29.029 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.010+0000 7fa5b1ffb700 1 --2- 172.21.15.6:0/1627484541 >> v2:172.21.15.22:6812/142603 conn(0x7fa5a00357a0 0x7fa5a0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.010+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa5b8047680 con 0x7fa5bc098670 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.011+0000 7fa5c3b24700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.011+0000 7fa5c3b24700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.011+0000 7fa5c3b24700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.011+0000 7fa5c10bf700 1 --2- 172.21.15.6:0/1627484541 >> v2:172.21.15.22:6812/142603 conn(0x7fa5a00357a0 0x7fa5a0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.011+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5bc09d940 con 0x7fa5bc098670 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.011+0000 7fa5c10bf700 10 monclient: get_auth_request con 0x7fa5a00357a0 auth_method 0 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.013+0000 7fa5c10bf700 1 --2- 172.21.15.6:0/1627484541 >> v2:172.21.15.22:6812/142603 conn(0x7fa5a00357a0 0x7fa5a0037c50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac003080 tx=0x7fa5ac016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.016+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa5b801c460 con 0x7fa5bc098670 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.016+0000 7fa5b1ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:29.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.016+0000 7fa5b1ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.218+0000 7fa5c3b24700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:29.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.218+0000 7fa5c3b24700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.218+0000 7fa5c3b24700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.218+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa5bc09e770 con 0x7fa5bc098670 2023-09-30T00:44:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.219+0000 7fa5b1ffb700 1 -- 172.21.15.6:0/1627484541 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v16) v1 ==== 74+0+9991 (secure 0 0 0) 0x7fa5b80102e0 con 0x7fa5bc098670 2023-09-30T00:44:29.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.219+0000 7fa5b1ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:29.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.219+0000 7fa5b1ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:29.220 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:29.220 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":16,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:28.406248+0000","last_up_change":"2023-09-30T00:44:23.223970+0000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:44:23.604669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:44:26.578400+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":114301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":114301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":114301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":114301}]},"public_addr":"172.21.15.6:6804/114301","cluster_addr":"172.21.15.6:6805/114301","heartbeat_back_addr":"172.21.15.6:6807/114301","heartbeat_front_addr":"172.21.15.6:6806/114301","state":["exists","up"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":114300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":114300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":114300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":114300}]},"public_addr":"172.21.15.6:6800/114300","cluster_addr":"172.21.15.6:6801/114300","heartbeat_back_addr":"172.21.15.6:6803/114300","heartbeat_front_addr":"172.21.15.6:6802/114300","state":["exists","up"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":114302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":114302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":114302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":114302}]},"public_addr":"172.21.15.6:6808/114302","cluster_addr":"172.21.15.6:6809/114302","heartbeat_back_addr":"172.21.15.6:6811/114302","heartbeat_front_addr":"172.21.15.6:6810/114302","state":["exists","up"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":143135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6801","nonce":143135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6803","nonce":143135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":143135}]},"public_addr":"172.21.15.22:6800/143135","cluster_addr":"172.21.15.22:6801/143135","heartbeat_back_addr":"172.21.15.22:6803/143135","heartbeat_front_addr":"172.21.15.22:6802/143135","state":["exists","up"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":143134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":143134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6811","nonce":143134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":143134}]},"public_addr":"172.21.15.22:6808/143134","cluster_addr":"172.21.15.22:6809/143134","heartbeat_back_addr":"172.21.15.22:6811/143134","heartbeat_front_addr":"172.21.15.22:6810/143134","state":["exists","up"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":143133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6805","nonce":143133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6807","nonce":143133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":143133}]},"public_addr":"172.21.15.22:6804/143133","cluster_addr":"172.21.15.22:6805/143133","heartbeat_back_addr":"172.21.15.22:6807/143133","heartbeat_front_addr":"172.21.15.22:6806/143133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.597432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.528045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.614505+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.340159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.641614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.578369+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:29.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 >> v2:172.21.15.22:6812/142603 conn(0x7fa5a00357a0 msgr2=0x7fa5a0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1627484541 >> v2:172.21.15.22:6812/142603 conn(0x7fa5a00357a0 0x7fa5a0037c50 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac003080 tx=0x7fa5ac016000 comp rx=0 tx=0).stop 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 10 monclient: shutdown 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 msgr2=0x7fa5bc12f3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 0x7fa5bc12f3a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800c5d0 tx=0x7fa5b8007f60 comp rx=0 tx=0).stop 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 shutdown_connections 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1627484541 >> v2:172.21.15.22:6812/142603 conn(0x7fa5a00357a0 0x7fa5a0037c50 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5bc137b60 0x7fa5bc09cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5bc098670 0x7fa5bc12f3a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 --2- 172.21.15.6:0/1627484541 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5bc132d00 0x7fa5bc09c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 >> 172.21.15.6:0/1627484541 conn(0x7fa5bc0a5480 msgr2=0x7fa5bc068ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 shutdown_connections 2023-09-30T00:44:29.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.221+0000 7fa5c3b24700 1 -- 172.21.15.6:0/1627484541 wait complete. 2023-09-30T00:44:29.234 INFO:tasks.ceph.ceph_manager.ceph:all up! 2023-09-30T00:44:29.235 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:44:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.365+0000 7f8d862e5700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.365+0000 7f8d862e5700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.365+0000 7f8d862e5700 10 monclient: monmap: 2023-09-30T00:44:29.365 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.365845+0000 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.365845+0000 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.366 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.366+0000 7f8d862e5700 1 Processor -- start 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.366+0000 7f8d862e5700 1 -- start start 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.366+0000 7f8d862e5700 10 monclient: init 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient: _add_conns ranks=[4,0,2,3,1] 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8d80098670 0x7f8d80132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): picked mon.noname-c con 0x7f8d80098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8d80098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:29.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 0x7f8d8012f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): picked mon.noname-b con 0x7f8d80098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d800a1000 con 0x7f8d80098a40 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d800a0a50 con 0x7f8d80098670 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800a0730 con 0x7f8d80098e90 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d862e5700 10 monclient(hunting): authenticate will time out at 1265.664161s 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7f7fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8d80098a40 0x7f8d8012ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60840/0 (socket says 172.21.15.6:60840) 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 0x7f8d8012f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7f7fe700 1 -- 172.21.15.6:0/1411286500 learned_addr learned my addr 172.21.15.6:0/1411286500 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): get_auth_request con 0x7f8d80098e90 auth_method 0 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 1 --2- 172.21.15.6:0/1411286500 >> v2:172.21.15.22:3300/0 conn(0x7f8d80098670 0x7f8d80132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): get_auth_request con 0x7f8d80098670 auth_method 0 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient(hunting): handle_auth_done global_id 4223 payload 274 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 1 -- 172.21.15.6:0/1411286500 >> v2:172.21.15.22:3300/0 conn(0x7f8d80098670 msgr2=0x7f8d80132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 1 --2- 172.21.15.6:0/1411286500 >> v2:172.21.15.22:3300/0 conn(0x7f8d80098670 0x7f8d80132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 1 -- 172.21.15.6:0/1411286500 >> v1:172.21.15.6:6789/0 conn(0x7f8d80098a40 legacy=0x7f8d8012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 1 monclient: found mon.noname-b 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.367+0000 7f8d84882700 1 -- 172.21.15.6:0/1411286500 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d80068df0 con 0x7f8d80098e90 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.369+0000 7f8d84882700 1 -- 172.21.15.6:0/1411286500 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 msgr2=0x7f8d8012f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.369+0000 7f8d84882700 1 --2- 172.21.15.6:0/1411286500 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 0x7f8d8012f4f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009e60 tx=0x7f8d78009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90b051f38ce9aad8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 1 -- 172.21.15.6:0/1411286500 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800e070 con 0x7f8d80098e90 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: dump: 2023-09-30T00:44:29.378 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 1 -- 172.21.15.6:0/1411286500 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d7800bd30 con 0x7f8d80098e90 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 5 monclient: authenticate success, global_id 4223 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 1 -- 172.21.15.6:0/1411286500 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d7800bea0 con 0x7f8d80098e90 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 4 set_mon_vals no callback set 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d7effd700 10 monclient: dump: 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.380 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 10 monclient: shutdown 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 -- 172.21.15.6:0/1411286500 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 msgr2=0x7f8d8012f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 --2- 172.21.15.6:0/1411286500 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 0x7f8d8012f4f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8d78009e60 tx=0x7f8d78009780 comp rx=0 tx=0).stop 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 -- 172.21.15.6:0/1411286500 shutdown_connections 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 --2- 172.21.15.6:0/1411286500 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098e90 0x7f8d8012f4f0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 --2- 172.21.15.6:0/1411286500 >> v2:172.21.15.22:3300/0 conn(0x7f8d80098670 0x7f8d80132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 -- 172.21.15.6:0/1411286500 >> 172.21.15.6:0/1411286500 conn(0x7f8d800a5480 msgr2=0x7f8d800a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 -- 172.21.15.6:0/1411286500 shutdown_connections 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.370+0000 7f8d862e5700 1 -- 172.21.15.6:0/1411286500 wait complete. 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.371+0000 7f8d862e5700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.371+0000 7f8d862e5700 10 monclient: monmap: 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.382 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.371+0000 7f8d862e5700 1 Processor -- start 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.371+0000 7f8d862e5700 1 -- start start 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.371+0000 7f8d862e5700 10 monclient: init 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8d80098670 0x7f8d80141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): picked mon.noname-b con 0x7f8d80098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 0x7f8d8009ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): picked mon.noname-c con 0x7f8d80098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 0x7f8d800994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): picked mon.noname-a con 0x7f8d80098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d8009d1c0 con 0x7f8d80098e90 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800a1500 con 0x7f8d80098670 2023-09-30T00:44:29.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d800a3a80 con 0x7f8d80098a40 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d862e5700 10 monclient(hunting): authenticate will time out at 1265.669510s 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 0x7f8d8009ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 0x7f8d800994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 0x7f8d8009ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36454/0 (socket says 172.21.15.6:36454) 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 0x7f8d800994d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41902/0 (socket says 172.21.15.6:41902) 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 -- 172.21.15.6:0/4216640515 learned_addr learned my addr 172.21.15.6:0/4216640515 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): get_auth_request con 0x7f8d80098a40 auth_method 0 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): get_auth_request con 0x7f8d80098e90 auth_method 0 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8d80098670 0x7f8d80141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): get_auth_request con 0x7f8d80098670 auth_method 0 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d84882700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient(hunting): handle_auth_done global_id 4226 payload 824 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 -- 172.21.15.6:0/4216640515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8d80098670 msgr2=0x7f8d80141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8d80098670 0x7f8d80141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 -- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 msgr2=0x7f8d800994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 0x7f8d800994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.372+0000 7f8d7f7fe700 1 -- 172.21.15.6:0/4216640515 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d78009a20 con 0x7f8d80098a40 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.373+0000 7f8d84882700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 0x7f8d800994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d7f7fe700 1 -- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 msgr2=0x7f8d8009ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d7f7fe700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 0x7f8d8009ebb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000c640 tx=0x7f8d70004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d70005d40 con 0x7f8d80098a40 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: dump: 2023-09-30T00:44:29.388 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d700070d0 con 0x7f8d80098a40 2023-09-30T00:44:29.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 5 monclient: authenticate success, global_id 4226 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 10 monclient: _renew_subs 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d8009af60 con 0x7f8d80098a40 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d70007490 con 0x7f8d80098a40 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d77fff700 10 monclient: dump: 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.390 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d7e7fc700 4 set_mon_vals no callback set 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 10 monclient: _renew_subs 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.375+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d8009bb10 con 0x7f8d80098a40 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8d7001b3b0 con 0x7f8d80098a40 2023-09-30T00:44:29.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d862e5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d862e5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d862e5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d77fff700 1 --2- 172.21.15.6:0/4216640515 >> v2:172.21.15.22:6812/142603 conn(0x7f8d600357a0 0x7f8d60037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d80072b00 con 0x7f8d80098a40 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8d70046df0 con 0x7f8d80098a40 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d7ffff700 1 --2- 172.21.15.6:0/4216640515 >> v2:172.21.15.22:6812/142603 conn(0x7f8d600357a0 0x7f8d60037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.376+0000 7f8d7ffff700 10 monclient: get_auth_request con 0x7f8d600357a0 auth_method 0 2023-09-30T00:44:29.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.379+0000 7f8d7ffff700 1 --2- 172.21.15.6:0/4216640515 >> v2:172.21.15.22:6812/142603 conn(0x7f8d600357a0 0x7f8d60037c50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009db0 tx=0x7f8d680091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.381+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8d7001a360 con 0x7f8d80098a40 2023-09-30T00:44:29.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.381+0000 7f8d77fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:29.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.381+0000 7f8d77fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.591+0000 7f8d862e5700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.591+0000 7f8d862e5700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.591+0000 7f8d862e5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.591+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d800a2bc0 con 0x7f8d80098a40 2023-09-30T00:44:29.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.592+0000 7f8d77fff700 1 -- 172.21.15.6:0/4216640515 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+9978 (secure 0 0 0) 0x7f8d70018e00 con 0x7f8d80098a40 2023-09-30T00:44:29.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.592+0000 7f8d77fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:44:29.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.592+0000 7f8d77fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:29.592 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:29.592 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:29.409969+0000","last_up_change":"2023-09-30T00:44:23.223970+0000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:44:23.604669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:44:26.578400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":114301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":114301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":114301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":114301}]},"public_addr":"172.21.15.6:6804/114301","cluster_addr":"172.21.15.6:6805/114301","heartbeat_back_addr":"172.21.15.6:6807/114301","heartbeat_front_addr":"172.21.15.6:6806/114301","state":["exists","up"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":114300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":114300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":114300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":114300}]},"public_addr":"172.21.15.6:6800/114300","cluster_addr":"172.21.15.6:6801/114300","heartbeat_back_addr":"172.21.15.6:6803/114300","heartbeat_front_addr":"172.21.15.6:6802/114300","state":["exists","up"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":114302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":114302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":114302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":114302}]},"public_addr":"172.21.15.6:6808/114302","cluster_addr":"172.21.15.6:6809/114302","heartbeat_back_addr":"172.21.15.6:6811/114302","heartbeat_front_addr":"172.21.15.6:6810/114302","state":["exists","up"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":143135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6801","nonce":143135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6803","nonce":143135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":143135}]},"public_addr":"172.21.15.22:6800/143135","cluster_addr":"172.21.15.22:6801/143135","heartbeat_back_addr":"172.21.15.22:6803/143135","heartbeat_front_addr":"172.21.15.22:6802/143135","state":["exists","up"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":143134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":143134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6811","nonce":143134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":143134}]},"public_addr":"172.21.15.22:6808/143134","cluster_addr":"172.21.15.22:6809/143134","heartbeat_back_addr":"172.21.15.22:6811/143134","heartbeat_front_addr":"172.21.15.22:6810/143134","state":["exists","up"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":143133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6805","nonce":143133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6807","nonce":143133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":143133}]},"public_addr":"172.21.15.22:6804/143133","cluster_addr":"172.21.15.22:6805/143133","heartbeat_back_addr":"172.21.15.22:6807/143133","heartbeat_front_addr":"172.21.15.22:6806/143133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.597432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.528045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.614505+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.340159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.641614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.578369+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:44:29.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 >> v2:172.21.15.22:6812/142603 conn(0x7f8d600357a0 msgr2=0x7f8d60037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 --2- 172.21.15.6:0/4216640515 >> v2:172.21.15.22:6812/142603 conn(0x7f8d600357a0 0x7f8d60037c50 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8d68009db0 tx=0x7f8d680091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 10 monclient: shutdown 2023-09-30T00:44:29.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 msgr2=0x7f8d8009ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 0x7f8d8009ebb0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f8d7000c640 tx=0x7f8d70004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 shutdown_connections 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 --2- 172.21.15.6:0/4216640515 >> v2:172.21.15.22:6812/142603 conn(0x7f8d600357a0 0x7f8d60037c50 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8d80098e90 0x7f8d800994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8d80098670 0x7f8d80141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 --2- 172.21.15.6:0/4216640515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8d80098a40 0x7f8d8009ebb0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 >> 172.21.15.6:0/4216640515 conn(0x7f8d800a5480 msgr2=0x7f8d80069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 shutdown_connections 2023-09-30T00:44:29.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.594+0000 7f8d862e5700 1 -- 172.21.15.6:0/4216640515 wait complete. 2023-09-30T00:44:29.607 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2023-09-30T00:44:29.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2023-09-30T00:44:29.608 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2023-09-30T00:44:29.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2023-09-30T00:44:29.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2023-09-30T00:44:29.609 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2023-09-30T00:44:29.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.765+0000 7faabfa85700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.765+0000 7faabfa85700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.765+0000 7faabfa85700 10 monclient: monmap: 2023-09-30T00:44:29.765 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.765 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.765 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.765459+0000 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.765459+0000 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.766 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.765+0000 7f0cc24cf700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.765+0000 7f0cc24cf700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.765+0000 7f0cc24cf700 10 monclient: monmap: 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.765538+0000 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.765538+0000 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.767 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.766+0000 7faabfa85700 1 Processor -- start 2023-09-30T00:44:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.766+0000 7f0cc24cf700 1 Processor -- start 2023-09-30T00:44:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.766+0000 7f0cc24cf700 1 -- start start 2023-09-30T00:44:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.766+0000 7f0cc24cf700 10 monclient: init 2023-09-30T00:44:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.766+0000 7faabfa85700 1 -- start start 2023-09-30T00:44:29.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.766+0000 7faabfa85700 10 monclient: init 2023-09-30T00:44:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T00:44:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0cbc09a960 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc135300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc09ad30 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient: _add_conns ranks=[1,4,2,3,0] 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7faab8098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7faab8132d00 0x7faab8137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): picked mon.noname-c con 0x7faab8132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8137b60 0x7faab8139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): picked mon.noname-b con 0x7faab8137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faab80a1000 con 0x7faab8098670 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faab8099150 con 0x7faab8132d00 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab80a0a50 con 0x7faab8137b60 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabfa85700 10 monclient(hunting): authenticate will time out at 1266.064443s 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0cbc135840 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0985e0 con 0x7f0cbc09a960 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cbc0993b0 con 0x7f0cbc135840 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc0990d0 con 0x7f0cbc09ad30 2023-09-30T00:44:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc24cf700 10 monclient(hunting): authenticate will time out at 1266.064364s 2023-09-30T00:44:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc0a6c700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f0cbc135840 0x7f0cbc137c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57272/0 (socket says 172.21.15.6:57272) 2023-09-30T00:44:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/2286724578 learned_addr learned my addr 172.21.15.6:0/2286724578 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:29.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd821700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7faab8098670 0x7faab8098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57278/0 (socket says 172.21.15.6:57278) 2023-09-30T00:44:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd821700 1 -- 172.21.15.6:0/698879591 learned_addr learned my addr 172.21.15.6:0/698879591 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd020700 1 --2- 172.21.15.6:0/698879591 >> v2:172.21.15.22:3300/0 conn(0x7faab8132d00 0x7faab8137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd020700 10 monclient(hunting): get_auth_request con 0x7faab8132d00 auth_method 0 2023-09-30T00:44:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd020700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd020700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.767+0000 7faabd020700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 1 --2- 172.21.15.6:0/2286724578 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc135300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): get_auth_request con 0x7f0cbc09ad30 auth_method 0 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.775 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.776 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabe022700 1 --2- 172.21.15.6:0/698879591 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8137b60 0x7faab8139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.776 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabe022700 10 monclient(hunting): get_auth_request con 0x7faab8137b60 auth_method 0 2023-09-30T00:44:29.776 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabe022700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.776 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabe022700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabe022700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_done global_id 4229 payload 274 2023-09-30T00:44:29.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 1 -- 172.21.15.6:0/2286724578 >> v1:172.21.15.22:6789/0 conn(0x7f0cbc135840 legacy=0x7f0cbc137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 1 -- 172.21.15.6:0/2286724578 >> v1:172.21.15.6:6789/0 conn(0x7f0cbc09a960 legacy=0x7f0cbc099d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 1 monclient: found mon.noname-b 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7f0cb3fff700 1 -- 172.21.15.6:0/2286724578 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cbc09cee0 con 0x7f0cbc09ad30 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 10 monclient(hunting): handle_auth_done global_id 4228 payload 274 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 1 -- 172.21.15.6:0/698879591 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8137b60 msgr2=0x7faab8139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 1 --2- 172.21.15.6:0/698879591 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8137b60 0x7faab8139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 1 -- 172.21.15.6:0/698879591 >> v1:172.21.15.22:6789/0 conn(0x7faab8098670 legacy=0x7faab8098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.768+0000 7faabd020700 1 -- 172.21.15.6:0/698879591 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab80a0730 con 0x7faab8132d00 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cb3fff700 1 -- 172.21.15.6:0/2286724578 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 msgr2=0x7f0cbc135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.779 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cb3fff700 1 --2- 172.21.15.6:0/2286724578 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc135300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009910 tx=0x7f0ca40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7033becfe99b5376 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabd020700 1 -- 172.21.15.6:0/698879591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 msgr2=0x7faab8137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabd020700 1 --2- 172.21.15.6:0/698879591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab8137620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faab4009910 tx=0x7faab40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8939bef7b76c3258 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 1 -- 172.21.15.6:0/698879591 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab400b9f0 con 0x7faab8132d00 2023-09-30T00:44:29.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: dump: 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.781 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 1 -- 172.21.15.6:0/698879591 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab400bb50 con 0x7faab8132d00 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabfa85700 5 monclient: authenticate success, global_id 4228 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabfa85700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 1 -- 172.21.15.6:0/698879591 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab4011420 con 0x7faab8132d00 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabc81f700 10 monclient: dump: 2023-09-30T00:44:29.782 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabfa85700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7faabfa85700 4 set_mon_vals no callback set 2023-09-30T00:44:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 1 -- 172.21.15.6:0/2286724578 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca400b9f0 con 0x7f0cbc09ad30 2023-09-30T00:44:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:29.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: dump: 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 1 -- 172.21.15.6:0/2286724578 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ca400bb50 con 0x7f0cbc09ad30 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 1 -- 172.21.15.6:0/2286724578 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ca4011420 con 0x7f0cbc09ad30 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cbb7fe700 10 monclient: dump: 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.786 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cc24cf700 5 monclient: authenticate success, global_id 4229 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cc24cf700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.771+0000 7f0cc24cf700 4 set_mon_vals no callback set 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 10 monclient: shutdown 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 -- 172.21.15.6:0/2286724578 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 msgr2=0x7f0cbc135300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 --2- 172.21.15.6:0/2286724578 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc135300 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0ca4009910 tx=0x7f0ca40091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 -- 172.21.15.6:0/2286724578 shutdown_connections 2023-09-30T00:44:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 --2- 172.21.15.6:0/2286724578 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc135300 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 -- 172.21.15.6:0/2286724578 >> 172.21.15.6:0/2286724578 conn(0x7f0cbc09ddb0 msgr2=0x7f0cbc09e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 10 monclient: shutdown 2023-09-30T00:44:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 -- 172.21.15.6:0/698879591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 msgr2=0x7faab8137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 --2- 172.21.15.6:0/698879591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab8137620 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7faab4009910 tx=0x7faab40091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 -- 172.21.15.6:0/2286724578 shutdown_connections 2023-09-30T00:44:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 -- 172.21.15.6:0/698879591 shutdown_connections 2023-09-30T00:44:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 --2- 172.21.15.6:0/698879591 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8137b60 0x7faab8139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 --2- 172.21.15.6:0/698879591 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab8137620 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 -- 172.21.15.6:0/698879591 >> 172.21.15.6:0/698879591 conn(0x7faab80a5480 msgr2=0x7faab80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7faabfa85700 1 -- 172.21.15.6:0/698879591 shutdown_connections 2023-09-30T00:44:29.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.773+0000 7faabfa85700 1 -- 172.21.15.6:0/698879591 wait complete. 2023-09-30T00:44:29.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.772+0000 7f0cc24cf700 1 -- 172.21.15.6:0/2286724578 wait complete. 2023-09-30T00:44:29.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.773+0000 7f0cc24cf700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.773+0000 7f0cc24cf700 10 monclient: monmap: 2023-09-30T00:44:29.791 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.791 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.773+0000 7f0cc24cf700 1 Processor -- start 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.773+0000 7f0cc24cf700 1 -- start start 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.773+0000 7f0cc24cf700 10 monclient: init 2023-09-30T00:44:29.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0cbc09a960 0x7f0cbc13d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): picked mon.noname-b con 0x7f0cbc09a960 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc134c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): picked mon.noname-c con 0x7f0cbc09ad30 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 0x7f0cbc138c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): picked mon.noname-a con 0x7f0cbc135840 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc09b3f0 con 0x7f0cbc135840 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc13d880 con 0x7f0cbc09a960 2023-09-30T00:44:29.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc13b440 con 0x7f0cbc09ad30 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cc24cf700 10 monclient(hunting): authenticate will time out at 1266.071316s 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7f0cbbfff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0cbc09a960 0x7f0cbc13d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 0x7f0cbc138c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 0x7f0cbc138c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41918/0 (socket says 172.21.15.6:41918) 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/4289296779 learned_addr learned my addr 172.21.15.6:0/4289296779 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): get_auth_request con 0x7f0cbc135840 auth_method 0 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 1 --2- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc134c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): get_auth_request con 0x7f0cbc09ad30 auth_method 0 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cb3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7faabfa85700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7faabfa85700 10 monclient: monmap: 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.796 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.774+0000 7faabfa85700 1 Processor -- start 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient(hunting): handle_auth_done global_id 4224 payload 824 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 msgr2=0x7f0cbc134c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 --2- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0cbc09ad30 0x7f0cbc134c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/4289296779 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0cbc09a960 msgr2=0x7f0cbc13d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 --2- 172.21.15.6:0/4289296779 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0cbc09a960 0x7f0cbc13d340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 monclient: found mon.noname-a 2023-09-30T00:44:29.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/4289296779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ca4009420 con 0x7f0cbc135840 2023-09-30T00:44:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7faabfa85700 1 -- start start 2023-09-30T00:44:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.775+0000 7faabfa85700 10 monclient: init 2023-09-30T00:44:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8098670 0x7faab809c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): picked mon.noname-c con 0x7faab8098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab809cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): picked mon.noname-b con 0x7faab8132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faab8137b60 0x7faab809d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): picked mon.noname-a con 0x7faab8137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab80a1500 con 0x7faab8137b60 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab81377f0 con 0x7faab8132d00 2023-09-30T00:44:29.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faab809f530 con 0x7faab8098670 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabfa85700 10 monclient(hunting): authenticate will time out at 1266.073507s 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab809cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd821700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8098670 0x7faab809c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab809cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.6:49260/0 (socket says 172.21.15.6:49260) 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 1 -- 172.21.15.6:0/3768965734 learned_addr learned my addr 172.21.15.6:0/3768965734 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 10 monclient(hunting): get_auth_request con 0x7faab8132d00 auth_method 0 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd020700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd821700 10 monclient(hunting): get_auth_request con 0x7faab8098670 auth_method 0 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd821700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd821700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.776+0000 7faabd821700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd020700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd020700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd020700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 10 monclient(hunting): handle_auth_done global_id 4238 payload 824 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabe022700 1 --2- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faab8137b60 0x7faab809d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 1 -- 172.21.15.6:0/3768965734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 msgr2=0x7faab809cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 1 --2- 172.21.15.6:0/3768965734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faab8132d00 0x7faab809cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 1 -- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faab8137b60 msgr2=0x7faab809d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 1 --2- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faab8137b60 0x7faab809d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabd821700 1 -- 172.21.15.6:0/3768965734 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab4009420 con 0x7faab8098670 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabe022700 10 monclient: get_auth_request con 0x7faab8137b60 auth_method 0 2023-09-30T00:44:29.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7faabe022700 1 --2- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faab8137b60 0x7faab809d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:44:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 msgr2=0x7f0cbc138c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.777+0000 7f0cc0a6c700 1 --2- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 0x7f0cbc138c20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400cbf0 tx=0x7f0cb40053d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb400ab20 con 0x7f0cbc135840 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: dump: 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.805 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb400aca0 con 0x7f0cbc135840 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cb4007340 con 0x7f0cbc135840 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cb8ff9700 10 monclient: dump: 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.806 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 5 monclient: authenticate success, global_id 4224 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cbaffd700 4 set_mon_vals no callback set 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 10 monclient: _renew_subs 2023-09-30T00:44:29.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc13c330 con 0x7f0cbc135840 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 10 monclient: _renew_subs 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.778+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc135160 con 0x7f0cbc135840 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7f0cc24cf700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0cbc072b00 con 0x7f0cbc135840 2023-09-30T00:44:29.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faabd821700 1 -- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8098670 msgr2=0x7faab809c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faabd821700 1 --2- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8098670 0x7faab809c7a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faab000cc10 tx=0x7faab00053c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab000aab0 con 0x7faab8098670 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faaadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faaadffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faaadffb700 10 monclient: dump: 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.809 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faaadffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faaadffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faabfa85700 5 monclient: authenticate success, global_id 4238 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faabfa85700 10 monclient: _renew_subs 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faabfa85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab809f780 con 0x7faab8098670 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.779+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0cb4007730 con 0x7f0cbc135840 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7f0cb8ff9700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.22:6812/142603 conn(0x7f0ca8035950 0x7f0ca8037e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0cb40488a0 con 0x7f0cbc135840 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7f0cb8ff9700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.6:6804/114301 conn(0x7f0ca803b4f0 0x7f0ca803d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 --> v2:172.21.15.6:6804/114301 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0ca803df70 con 0x7f0ca803b4f0 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cb4048b40 con 0x7f0cbc135840 2023-09-30T00:44:29.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7f0cb8ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:44:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faab000ac30 con 0x7faab8098670 2023-09-30T00:44:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaadffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faab00070f0 con 0x7faab8098670 2023-09-30T00:44:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaadffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaadffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaadffb700 10 monclient: dump: 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faabfa85700 10 monclient: _renew_subs 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faabfa85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab812f730 con 0x7faab8098670 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.780+0000 7faaaf7fe700 4 set_mon_vals no callback set 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.781+0000 7faabfa85700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.781+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7faab8072b00 con 0x7faab8098670 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.781+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7faab0007500 con 0x7faab8098670 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faaadffb700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6812/142603 conn(0x7faa9c0357a0 0x7faa9c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7faab0048790 con 0x7faab8098670 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faaadffb700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6800/143135 conn(0x7faa9c03b2c0 0x7faa9c03d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 --> v2:172.21.15.22:6800/143135 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7faa9c03dd40 con 0x7faa9c03b2c0 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7faab0048a30 con 0x7faab8098670 2023-09-30T00:44:29.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faaadffb700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faabd020700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6812/142603 conn(0x7faa9c0357a0 0x7faa9c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faabe022700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6800/143135 conn(0x7faa9c03b2c0 0x7faa9c03d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faabd020700 10 monclient: get_auth_request con 0x7faa9c0357a0 auth_method 0 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.782+0000 7faabe022700 10 monclient: get_auth_request con 0x7faa9c03b2c0 auth_method 0 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.783+0000 7f0cbbfff700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.22:6812/142603 conn(0x7f0ca8035950 0x7f0ca8037e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.783+0000 7f0cbbfff700 10 monclient: get_auth_request con 0x7f0ca8035950 auth_method 0 2023-09-30T00:44:29.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.785+0000 7f0cb3fff700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.6:6804/114301 conn(0x7f0ca803b4f0 0x7f0ca803d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.785+0000 7f0cb3fff700 10 monclient: get_auth_request con 0x7f0ca803b4f0 auth_method 0 2023-09-30T00:44:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.785+0000 7faabd020700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6812/142603 conn(0x7faa9c0357a0 0x7faa9c037c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faab806ecf0 tx=0x7faab4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.785+0000 7f0cbbfff700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.22:6812/142603 conn(0x7f0ca8035950 0x7f0ca8037e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cac006fd0 tx=0x7f0cac008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.787+0000 7faabe022700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6800/143135 conn(0x7faa9c03b2c0 0x7faa9c03d6d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.788+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== osd.3 v2:172.21.15.22:6800/143135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7faa9c03dd40 con 0x7faa9c03b2c0 2023-09-30T00:44:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.792+0000 7f0cb3fff700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.6:6804/114301 conn(0x7f0ca803b4f0 0x7f0ca803d900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.793+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== osd.0 v2:172.21.15.6:6804/114301 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f0ca803df70 con 0x7f0ca803b4f0 2023-09-30T00:44:29.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.815+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 --> v2:172.21.15.22:6800/143135 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7faab8097620 con 0x7faa9c03b2c0 2023-09-30T00:44:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faaadffb700 1 -- 172.21.15.6:0/3768965734 <== osd.3 v2:172.21.15.22:6800/143135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7faab8097620 con 0x7faa9c03b2c0 2023-09-30T00:44:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6800/143135 conn(0x7faa9c03b2c0 msgr2=0x7faa9c03d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6800/143135 conn(0x7faa9c03b2c0 0x7faa9c03d6d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6812/142603 conn(0x7faa9c0357a0 msgr2=0x7faa9c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 --2- 172.21.15.6:0/3768965734 >> v2:172.21.15.22:6812/142603 conn(0x7faa9c0357a0 0x7faa9c037c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7faab806ecf0 tx=0x7faab4016000 comp rx=0 tx=0).stop 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 10 monclient: shutdown 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8098670 msgr2=0x7faab809c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 --2- 172.21.15.6:0/3768965734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faab8098670 0x7faab809c7a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7faab000cc10 tx=0x7faab00053c0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabe022700 1 -- 172.21.15.6:0/3768965734 reap_dead start 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 shutdown_connections 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.816+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 >> 172.21.15.6:0/3768965734 conn(0x7faab80a5480 msgr2=0x7faab81319e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.819+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 shutdown_connections 2023-09-30T00:44:29.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.819+0000 7faabfa85700 1 -- 172.21.15.6:0/3768965734 wait complete. 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.819+0000 7f69d0e0c700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.819+0000 7f69d0e0c700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.819+0000 7f69d0e0c700 10 monclient: monmap: 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.819895+0000 2023-09-30T00:44:29.820 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.819895+0000 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.820+0000 7f69d0e0c700 1 Processor -- start 2023-09-30T00:44:29.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.820+0000 7eff00b13700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.820+0000 7eff00b13700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.820+0000 7eff00b13700 10 monclient: monmap: 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.820763+0000 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.820763+0000 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.822 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.823 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.820+0000 7f69d0e0c700 1 -- start start 2023-09-30T00:44:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.820+0000 7f69d0e0c700 10 monclient: init 2023-09-30T00:44:29.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7eff00b13700 1 Processor -- start 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient: _add_conns ranks=[4,3,1,0,2] 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f69cc098670 0x7f69cc132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): picked mon.noname-c con 0x7f69cc098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f69cc098a40 0x7f69cc12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): picked mon.noname-a con 0x7f69cc098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f69cc098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69cc0a1000 con 0x7f69cc098e90 2023-09-30T00:44:29.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc0a0a50 con 0x7f69cc098a40 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69cc0a0730 con 0x7f69cc098670 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7f69d0e0c700 10 monclient(hunting): authenticate will time out at 1266.118766s 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7eff00b13700 1 -- start start 2023-09-30T00:44:29.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.821+0000 7eff00b13700 10 monclient: init 2023-09-30T00:44:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f69cc098a40 0x7f69cc12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f69cc098a40 0x7f69cc12ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41944/0 (socket says 172.21.15.6:41944) 2023-09-30T00:44:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 -- 172.21.15.6:0/3531168484 learned_addr learned my addr 172.21.15.6:0/3531168484 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): get_auth_request con 0x7f69cc098a40 auth_method 0 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient(hunting): handle_auth_done global_id 4227 payload 274 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 -- 172.21.15.6:0/3531168484 >> v2:172.21.15.22:3300/0 conn(0x7f69cc098670 msgr2=0x7f69cc132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 --2- 172.21.15.6:0/3531168484 >> v2:172.21.15.22:3300/0 conn(0x7f69cc098670 0x7f69cc132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 -- 172.21.15.6:0/3531168484 >> v1:172.21.15.22:6789/0 conn(0x7f69cc098e90 legacy=0x7f69cc12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 monclient: found mon.noname-a 2023-09-30T00:44:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7f69c9d9b700 1 -- 172.21.15.6:0/3531168484 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69cc068df0 con 0x7f69cc098a40 2023-09-30T00:44:29.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient: _add_conns ranks=[1,2,3,0,4] 2023-09-30T00:44:29.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7efefc09cd40 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:29.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc069730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): picked mon.noname-b con 0x7efefc09a8f0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efefc13b6c0 0x7efefc13daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): picked mon.noname-a con 0x7efefc13b6c0 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efefc068820 con 0x7efefc09cd40 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efefc06a0b0 con 0x7efefc13b6c0 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc09f090 con 0x7efefc09a8f0 2023-09-30T00:44:29.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.822+0000 7eff00b13700 10 monclient(hunting): authenticate will time out at 1266.119743s 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efefc13b6c0 0x7efefc13daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7efefc13b6c0 0x7efefc13daa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41946/0 (socket says 172.21.15.6:41946) 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 1 -- 172.21.15.6:0/346538531 learned_addr learned my addr 172.21.15.6:0/346538531 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): get_auth_request con 0x7efefc13b6c0 auth_method 0 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 1 --2- 172.21.15.6:0/346538531 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): get_auth_request con 0x7efefc09a8f0 auth_method 0 2023-09-30T00:44:29.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efefad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient(hunting): handle_auth_done global_id 4241 payload 274 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 1 -- 172.21.15.6:0/346538531 >> v2:172.21.15.6:3300/0 conn(0x7efefc13b6c0 msgr2=0x7efefc13daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 1 --2- 172.21.15.6:0/346538531 >> v2:172.21.15.6:3300/0 conn(0x7efefc13b6c0 0x7efefc13daa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 1 -- 172.21.15.6:0/346538531 >> v1:172.21.15.22:6789/0 conn(0x7efefc09cd40 legacy=0x7efefc09d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 1 monclient: found mon.noname-b 2023-09-30T00:44:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.823+0000 7efef9d9b700 1 -- 172.21.15.6:0/346538531 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efefc06a6e0 con 0x7efefc09a8f0 2023-09-30T00:44:29.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c9d9b700 1 -- 172.21.15.6:0/3531168484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098a40 msgr2=0x7f69cc12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c9d9b700 1 --2- 172.21.15.6:0/3531168484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098a40 0x7f69cc12ede0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f69cc09fe60 tx=0x7f69c0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e507997197bb2c5f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 1 -- 172.21.15.6:0/3531168484 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c000ba30 con 0x7f69cc098a40 2023-09-30T00:44:29.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: dump: 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.835 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69d0e0c700 5 monclient: authenticate success, global_id 4227 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69d0e0c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 1 -- 172.21.15.6:0/3531168484 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c000bb90 con 0x7f69cc098a40 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69d0e0c700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 1 -- 172.21.15.6:0/3531168484 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c0011420 con 0x7f69cc098a40 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69d0e0c700 4 set_mon_vals no callback set 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:29.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.825+0000 7f69c959a700 10 monclient: dump: 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.837 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 --> v2:172.21.15.6:6804/114301 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0cbc09d160 con 0x7f0ca803b4f0 2023-09-30T00:44:29.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 10 monclient: shutdown 2023-09-30T00:44:29.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 1 -- 172.21.15.6:0/3531168484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098a40 msgr2=0x7f69cc12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 1 --2- 172.21.15.6:0/3531168484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098a40 0x7f69cc12ede0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f69cc09fe60 tx=0x7f69c0009250 comp rx=0 tx=0).stop 2023-09-30T00:44:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f0cb8ff9700 1 -- 172.21.15.6:0/4289296779 <== osd.0 v2:172.21.15.6:6804/114301 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f0cbc09d160 con 0x7f0ca803b4f0 2023-09-30T00:44:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef9d9b700 1 -- 172.21.15.6:0/346538531 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 msgr2=0x7efefc069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef9d9b700 1 --2- 172.21.15.6:0/346538531 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc069730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efeec0098e0 tx=0x7efeec0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6252272e7603a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 1 -- 172.21.15.6:0/346538531 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec011840 con 0x7efefc09a8f0 2023-09-30T00:44:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: dump: 2023-09-30T00:44:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.840 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 1 -- 172.21.15.6:0/346538531 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeec0119a0 con 0x7efefc09a8f0 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7eff00b13700 5 monclient: authenticate success, global_id 4241 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7eff00b13700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:29.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7eff00b13700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 1 -- 172.21.15.6:0/346538531 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efeec011d40 con 0x7efefc09a8f0 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7eff00b13700 4 set_mon_vals no callback set 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7efef959a700 10 monclient: dump: 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.842 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 1 -- 172.21.15.6:0/3531168484 shutdown_connections 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 1 --2- 172.21.15.6:0/3531168484 >> v2:172.21.15.22:3300/0 conn(0x7f69cc098670 0x7f69cc132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 1 --2- 172.21.15.6:0/3531168484 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098a40 0x7f69cc12ede0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.826+0000 7f69d0e0c700 1 -- 172.21.15.6:0/3531168484 >> 172.21.15.6:0/3531168484 conn(0x7f69cc0a5480 msgr2=0x7f69cc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 >> v2:172.21.15.6:6804/114301 conn(0x7f0ca803b4f0 msgr2=0x7f0ca803d900 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.6:6804/114301 conn(0x7f0ca803b4f0 0x7f0ca803d900 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 >> v2:172.21.15.22:6812/142603 conn(0x7f0ca8035950 msgr2=0x7f0ca8037e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 1 --2- 172.21.15.6:0/4289296779 >> v2:172.21.15.22:6812/142603 conn(0x7f0ca8035950 0x7f0ca8037e00 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f0cac006fd0 tx=0x7f0cac008040 comp rx=0 tx=0).stop 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 10 monclient: shutdown 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 msgr2=0x7f0cbc138c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc24cf700 1 --2- 172.21.15.6:0/4289296779 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0cbc135840 0x7f0cbc138c20 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0cb400cbf0 tx=0x7f0cb40053d0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7f0cc0a6c700 1 -- 172.21.15.6:0/4289296779 reap_dead start 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7fa1da08b700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7fa1da08b700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7fa1da08b700 10 monclient: monmap: 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.845 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.827782+0000 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.827782+0000 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.846 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 10 monclient: shutdown 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 -- 172.21.15.6:0/346538531 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 msgr2=0x7efefc069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 --2- 172.21.15.6:0/346538531 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc069730 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7efeec0098e0 tx=0x7efeec0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 -- 172.21.15.6:0/346538531 shutdown_connections 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 --2- 172.21.15.6:0/346538531 >> v2:172.21.15.6:3300/0 conn(0x7efefc13b6c0 0x7efefc13daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 --2- 172.21.15.6:0/346538531 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc069730 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 -- 172.21.15.6:0/346538531 >> 172.21.15.6:0/346538531 conn(0x7efefc0a32b0 msgr2=0x7efefc0a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.827+0000 7eff00b13700 1 -- 172.21.15.6:0/346538531 shutdown_connections 2023-09-30T00:44:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7eff00b13700 1 -- 172.21.15.6:0/346538531 wait complete. 2023-09-30T00:44:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f69d0e0c700 1 -- 172.21.15.6:0/3531168484 shutdown_connections 2023-09-30T00:44:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f69d0e0c700 1 -- 172.21.15.6:0/3531168484 wait complete. 2023-09-30T00:44:29.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f2afae14700 10 monclient: get_monmap_and_config 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f2afae14700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f2afae14700 10 monclient: monmap: 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:29.828632+0000 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:29.828632+0000 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:29.849 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:29.850 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 shutdown_connections 2023-09-30T00:44:29.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 >> 172.21.15.6:0/4289296779 conn(0x7f0cbc09ddb0 msgr2=0x7f0cbc06d900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7eff00b13700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7eff00b13700 10 monclient: monmap: 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.851 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 Processor -- start 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 -- start start 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient: init 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f69d0e0c700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.828+0000 7f69d0e0c700 10 monclient: monmap: 2023-09-30T00:44:29.852 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7f69d0e0c700 1 Processor -- start 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc11c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): picked mon.noname-c con 0x7efefc09a8f0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efefc09cd40 0x7efefc13a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): picked mon.noname-b con 0x7efefc09cd40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 0x7efefc13ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): picked mon.noname-a con 0x7efefc13b6c0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc11bc30 con 0x7efefc13b6c0 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efefc06f1d0 con 0x7efefc09cd40 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efefc0a2e00 con 0x7efefc09a8f0 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7eff00b13700 10 monclient(hunting): authenticate will time out at 1266.126722s 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 shutdown_connections 2023-09-30T00:44:29.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7f0cc24cf700 1 -- 172.21.15.6:0/4289296779 wait complete. 2023-09-30T00:44:29.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 0x7efefc13ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 0x7efefc13ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41950/0 (socket says 172.21.15.6:41950) 2023-09-30T00:44:29.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 1 -- 172.21.15.6:0/123673512 learned_addr learned my addr 172.21.15.6:0/123673512 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): get_auth_request con 0x7efefc13b6c0 auth_method 0 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 1 --2- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc11c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): get_auth_request con 0x7efefc09a8f0 auth_method 0 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efefa59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7f69d0e0c700 1 -- start start 2023-09-30T00:44:29.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7f69d0e0c700 10 monclient: init 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 0x7f69cc12da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): picked mon.noname-c con 0x7f69cc098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69cc098a40 0x7f69cc12df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): picked mon.noname-b con 0x7f69cc098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098e90 0x7f69cc12e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): picked mon.noname-a con 0x7f69cc098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc0a1500 con 0x7f69cc098e90 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc135390 con 0x7f69cc098a40 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69cc09d770 con 0x7f69cc098670 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7f69d0e0c700 10 monclient(hunting): authenticate will time out at 1266.127712s 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 0x7f69cc12da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 0x7f69cc12da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36518/0 (socket says 172.21.15.6:36518) 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 -- 172.21.15.6:0/2852345781 learned_addr learned my addr 172.21.15.6:0/2852345781 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): get_auth_request con 0x7f69cc098670 auth_method 0 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7efef9d9b700 1 --2- 172.21.15.6:0/123673512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efefc09cd40 0x7efefc13a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efef9d9b700 10 monclient(hunting): get_auth_request con 0x7efefc09cd40 auth_method 0 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efef9d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efef9d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efef9d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 10 monclient(hunting): handle_auth_done global_id 4233 payload 824 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 1 -- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 msgr2=0x7efefc11c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 1 --2- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7efefc09a8f0 0x7efefc11c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 1 -- 172.21.15.6:0/123673512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efefc09cd40 msgr2=0x7efefc13a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 1 --2- 172.21.15.6:0/123673512 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7efefc09cd40 0x7efefc13a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 1 monclient: found mon.noname-a 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7efefad9d700 1 -- 172.21.15.6:0/123673512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeec009420 con 0x7efefc13b6c0 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 1 Processor -- start 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 1 -- start start 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient: init 2023-09-30T00:44:29.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa1d40631f0 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa1d406c660 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa1d406e3e0 0x7fa1d413f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient(hunting): picked mon.noname-c con 0x7fa1d406e3e0 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d40635c0 con 0x7fa1d40631f0 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1d406b6b0 con 0x7fa1d406c660 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1d40a0dd0 con 0x7fa1d406e3e0 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1da08b700 10 monclient(hunting): authenticate will time out at 1266.126736s 2023-09-30T00:44:29.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.829+0000 7fa1d3fff700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa1d406e3e0 0x7fa1d413f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d2ffd700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa1d406c660 0x7fa1d413bbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57306/0 (socket says 172.21.15.6:57306) 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d2ffd700 1 -- 172.21.15.6:0/2006357903 learned_addr learned my addr 172.21.15.6:0/2006357903 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): get_auth_request con 0x7fa1d406e3e0 auth_method 0 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_done global_id 4234 payload 274 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2006357903 >> v1:172.21.15.22:6789/0 conn(0x7fa1d406c660 legacy=0x7fa1d413bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2006357903 >> v1:172.21.15.6:6789/0 conn(0x7fa1d40631f0 legacy=0x7fa1d406dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.830+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2006357903 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d406b3d0 con 0x7fa1d406e3e0 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 1 --2- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098e90 0x7f69cc12e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): get_auth_request con 0x7f69cc098e90 auth_method 0 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69cad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient(hunting): handle_auth_done global_id 4247 payload 824 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 -- 172.21.15.6:0/2852345781 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69cc098a40 msgr2=0x7f69cc12df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 --2- 172.21.15.6:0/2852345781 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f69cc098a40 0x7f69cc12df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 -- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098e90 msgr2=0x7f69cc12e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 --2- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f69cc098e90 0x7f69cc12e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.831+0000 7f69ca59c700 1 -- 172.21.15.6:0/2852345781 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69c00094f0 con 0x7f69cc098670 2023-09-30T00:44:29.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 1 Processor -- start 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 1 -- start start 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient: init 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient: _add_conns ranks=[0,4,1,2,3] 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2af409d250 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2006357903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 msgr2=0x7fa1d413f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d3fff700 1 --2- 172.21.15.6:0/2006357903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d413f2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009e60 tx=0x7fa1cc009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76bb9fbdd4fc28f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f2af4132b60 0x7f2af4134f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient(hunting): picked mon.noname-c con 0x7f2af4132b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2af4135480 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af409c3a0 con 0x7f2af409d250 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2af409c0c0 con 0x7f2af4135480 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2af409bde0 con 0x7f2af4132b60 2023-09-30T00:44:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2afae14700 10 monclient(hunting): authenticate will time out at 1266.129781s 2023-09-30T00:44:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d27fc700 1 -- 172.21.15.6:0/2006357903 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc00e070 con 0x7fa1d406e3e0 2023-09-30T00:44:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d27fc700 10 monclient: dump: 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d27fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1da08b700 5 monclient: authenticate success, global_id 4234 2023-09-30T00:44:29.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7fa1da08b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2af93b1700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f2af4135480 0x7f2af4137860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57320/0 (socket says 172.21.15.6:57320) 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.832+0000 7f2af93b1700 1 -- 172.21.15.6:0/1706431098 learned_addr learned my addr 172.21.15.6:0/1706431098 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 1 --2- 172.21.15.6:0/1706431098 >> v2:172.21.15.22:3300/0 conn(0x7f2af4132b60 0x7f2af4134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): get_auth_request con 0x7f2af4132b60 auth_method 0 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient(hunting): handle_auth_done global_id 4240 payload 274 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 1 -- 172.21.15.6:0/1706431098 >> v1:172.21.15.22:6789/0 conn(0x7f2af4135480 legacy=0x7f2af4137860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 1 -- 172.21.15.6:0/1706431098 >> v1:172.21.15.6:6789/0 conn(0x7f2af409d250 legacy=0x7f2af409caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 1 monclient: found mon.noname-c 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7f2af3fff700 1 -- 172.21.15.6:0/1706431098 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af409abd0 con 0x7f2af4132b60 2023-09-30T00:44:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efefad9d700 1 -- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 msgr2=0x7efefc13ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efefad9d700 1 --2- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 0x7efefc13ae40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efef4009eb0 tx=0x7efef400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef400cc20 con 0x7efefc13b6c0 2023-09-30T00:44:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 10 monclient: dump: 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef400f040 con 0x7efefc13b6c0 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efef4014440 con 0x7efefc13b6c0 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efeea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7efeea7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7efeea7fc700 10 monclient: dump: 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.877 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.833+0000 7efef8d99700 4 set_mon_vals no callback set 2023-09-30T00:44:29.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ca59c700 1 -- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 msgr2=0x7f69cc12da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ca59c700 1 --2- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 0x7f69cc12da50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69c400bc30 tx=0x7f69c4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c40092e0 con 0x7f69cc098670 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: dump: 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.879 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f69c400c900 con 0x7f69cc098670 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69c400ccc0 con 0x7f69cc098670 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69ba7fc700 10 monclient: dump: 2023-09-30T00:44:29.880 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69c8d99700 4 set_mon_vals no callback set 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7eff00b13700 5 monclient: authenticate success, global_id 4233 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7eff00b13700 10 monclient: _renew_subs 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7eff00b13700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1d27fc700 1 -- 172.21.15.6:0/2006357903 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc012430 con 0x7fa1d406e3e0 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1d27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1d27fc700 1 -- 172.21.15.6:0/2006357903 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc0127d0 con 0x7fa1d406e3e0 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1d27fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:29.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1d27fc700 10 monclient: dump: 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1da08b700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7fa1da08b700 4 set_mon_vals no callback set 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7fa1da08b700 10 monclient: shutdown 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7fa1da08b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7fa1da08b700 1 -- 172.21.15.6:0/2006357903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 msgr2=0x7fa1d413f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7fa1da08b700 1 --2- 172.21.15.6:0/2006357903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d413f2e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa1cc009e60 tx=0x7fa1cc009780 comp rx=0 tx=0).stop 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7f2af3fff700 1 -- 172.21.15.6:0/1706431098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4132b60 msgr2=0x7f2af4134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7f2af3fff700 1 --2- 172.21.15.6:0/1706431098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4132b60 0x7f2af4134f40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009910 tx=0x7f2ae80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c90ca0ca13f1371b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 1 -- 172.21.15.6:0/1706431098 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae800b9f0 con 0x7f2af4132b60 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: dump: 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 1 -- 172.21.15.6:0/1706431098 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ae800bb50 con 0x7f2af4132b60 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 1 -- 172.21.15.6:0/1706431098 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ae8011420 con 0x7f2af4132b60 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2af37fe700 10 monclient: dump: 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.886 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 5 monclient: authenticate success, global_id 4240 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 4 set_mon_vals no callback set 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 5 monclient: authenticate success, global_id 4247 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 10 monclient: _renew_subs 2023-09-30T00:44:29.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69cc09dd40 con 0x7f69cc098670 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 10 monclient: _renew_subs 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69cc12f550 con 0x7f69cc098670 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7f69d0e0c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f69cc072b00 con 0x7f69cc098670 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efefc11cb50 con 0x7efefc13b6c0 2023-09-30T00:44:29.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.834+0000 7eff00b13700 10 monclient: _renew_subs 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7eff00b13700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efefc11b640 con 0x7efefc13b6c0 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7eff00b13700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.835+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7efefc072b00 con 0x7efefc13b6c0 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7fa1da08b700 1 -- 172.21.15.6:0/2006357903 shutdown_connections 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7fa1da08b700 1 --2- 172.21.15.6:0/2006357903 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d413f2e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7fa1da08b700 1 -- 172.21.15.6:0/2006357903 >> 172.21.15.6:0/2006357903 conn(0x7fa1d40680c0 msgr2=0x7fa1d409e450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f69c4012070 con 0x7f69cc098670 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69ba7fc700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6812/142603 conn(0x7f69ac0357f0 0x7f69ac037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f69c4047540 con 0x7f69cc098670 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69ba7fc700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6808/143134 conn(0x7f69ac03b390 0x7f69ac03d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 --> v2:172.21.15.22:6808/143134 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f69ac03de10 con 0x7f69ac03b390 2023-09-30T00:44:29.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69c4049a70 con 0x7f69cc098670 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69ba7fc700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69cad9d700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6808/143134 conn(0x7f69ac03b390 0x7f69ac03d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f69cad9d700 10 monclient: get_auth_request con 0x7f69ac03b390 auth_method 0 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f69c9d9b700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6812/142603 conn(0x7f69ac0357f0 0x7f69ac037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f69c9d9b700 10 monclient: get_auth_request con 0x7f69ac0357f0 auth_method 0 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7efef4014850 con 0x7efefc13b6c0 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7efeea7fc700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6812/142603 conn(0x7efedc0357f0 0x7efedc037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7efef4048ea0 con 0x7efefc13b6c0 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efeea7fc700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6804/143133 conn(0x7efedc03b390 0x7efedc03d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 --> v2:172.21.15.22:6804/143133 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7efedc03de10 con 0x7efedc03b390 2023-09-30T00:44:29.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7efef4049140 con 0x7efefc13b6c0 2023-09-30T00:44:29.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efeea7fc700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:44:29.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efefa59c700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6812/142603 conn(0x7efedc0357f0 0x7efedc037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efefa59c700 10 monclient: get_auth_request con 0x7efedc0357f0 auth_method 0 2023-09-30T00:44:29.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efef9d9b700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6804/143133 conn(0x7efedc03b390 0x7efedc03d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7efef9d9b700 10 monclient: get_auth_request con 0x7efedc03b390 auth_method 0 2023-09-30T00:44:29.894 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:44:29.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7fa1da08b700 1 -- 172.21.15.6:0/2006357903 shutdown_connections 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 10 monclient: shutdown 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 1 -- 172.21.15.6:0/1706431098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4132b60 msgr2=0x7f2af4134f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.836+0000 7f2afae14700 1 --2- 172.21.15.6:0/1706431098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4132b60 0x7f2af4134f40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2ae8009910 tx=0x7f2ae80091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f2afae14700 1 -- 172.21.15.6:0/1706431098 shutdown_connections 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f2afae14700 1 --2- 172.21.15.6:0/1706431098 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4132b60 0x7f2af4134f40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f2afae14700 1 -- 172.21.15.6:0/1706431098 >> 172.21.15.6:0/1706431098 conn(0x7f2af40a3320 msgr2=0x7f2af40a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f2afae14700 1 -- 172.21.15.6:0/1706431098 shutdown_connections 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.837+0000 7f2afae14700 1 -- 172.21.15.6:0/1706431098 wait complete. 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f2afae14700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f2afae14700 10 monclient: monmap: 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.896 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7fa1da08b700 1 -- 172.21.15.6:0/2006357903 wait complete. 2023-09-30T00:44:29.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f2afae14700 1 Processor -- start 2023-09-30T00:44:29.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f2afae14700 1 -- start start 2023-09-30T00:44:29.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7fa1da08b700 10 monclient: build_initial_monmap 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7f2afae14700 10 monclient: init 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.838+0000 7fa1da08b700 10 monclient: monmap: 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:29.898 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2af409d250 0x7f2af4099050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): picked mon.noname-c con 0x7f2af409d250 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af4132b60 0x7f2af406d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7fa1da08b700 1 Processor -- start 2023-09-30T00:44:29.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): picked mon.noname-a con 0x7f2af4132b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 0x7f2af406d580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): picked mon.noname-b con 0x7f2af4135480 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af406e5b0 con 0x7f2af4132b60 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2af406f620 con 0x7f2af4135480 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2af4096a90 con 0x7f2af409d250 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7f2afae14700 10 monclient(hunting): authenticate will time out at 1266.136724s 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7fa1da08b700 1 -- start start 2023-09-30T00:44:29.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.839+0000 7fa1da08b700 10 monclient: init 2023-09-30T00:44:29.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af4132b60 0x7f2af406d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 0x7f2af406d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 0x7f2af406d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.6:49286/0 (socket says 172.21.15.6:49286) 2023-09-30T00:44:29.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 -- 172.21.15.6:0/63221511 learned_addr learned my addr 172.21.15.6:0/63221511 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): get_auth_request con 0x7f2af4135480 auth_method 0 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af4132b60 0x7f2af406d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41966/0 (socket says 172.21.15.6:41966) 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): get_auth_request con 0x7f2af4132b60 auth_method 0 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7efefa59c700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6812/142603 conn(0x7efedc0357f0 0x7efedc037ca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efef00039e0 tx=0x7efef0007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af8bb0700 1 --2- 172.21.15.6:0/63221511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2af409d250 0x7f2af4099050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af8bb0700 10 monclient(hunting): get_auth_request con 0x7f2af409d250 auth_method 0 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af8bb0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af8bb0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af8bb0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient(hunting): handle_auth_done global_id 4243 payload 824 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 -- 172.21.15.6:0/63221511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2af409d250 msgr2=0x7f2af4099050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 --2- 172.21.15.6:0/63221511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2af409d250 0x7f2af4099050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 -- 172.21.15.6:0/63221511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af4132b60 msgr2=0x7f2af406d040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 --2- 172.21.15.6:0/63221511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2af4132b60 0x7f2af406d040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 monclient: found mon.noname-b 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:29.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1d40631f0 0x7fa1d413d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7f2af93b1700 1 -- 172.21.15.6:0/63221511 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae8009420 con 0x7f2af4135480 2023-09-30T00:44:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): picked mon.noname-a con 0x7fa1d40631f0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa1d406c660 0x7fa1d411bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): picked mon.noname-c con 0x7fa1d406c660 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d4139b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): picked mon.noname-b con 0x7fa1d406e3e0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d406eb00 con 0x7fa1d40631f0 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d411c880 con 0x7fa1d406e3e0 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d409c8b0 con 0x7fa1d406c660 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1da08b700 10 monclient(hunting): authenticate will time out at 1266.137408s 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d4139b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d4139b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.6:49288/0 (socket says 172.21.15.6:49288) 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2483332293 learned_addr learned my addr 172.21.15.6:0/2483332293 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:29.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): get_auth_request con 0x7fa1d406e3e0 auth_method 0 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 1 --2- 172.21.15.6:0/2483332293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1d40631f0 0x7fa1d413d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): get_auth_request con 0x7fa1d40631f0 auth_method 0 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 1 --2- 172.21.15.6:0/2483332293 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa1d406c660 0x7fa1d411bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): get_auth_request con 0x7fa1d406c660 auth_method 0 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.840+0000 7fa1d3fff700 10 monclient(hunting): handle_auth_done global_id 4246 payload 824 2023-09-30T00:44:29.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2483332293 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa1d406c660 msgr2=0x7fa1d411bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 1 --2- 172.21.15.6:0/2483332293 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa1d406c660 0x7fa1d411bb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2483332293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1d40631f0 msgr2=0x7fa1d413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 1 --2- 172.21.15.6:0/2483332293 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa1d40631f0 0x7fa1d413d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 1 monclient: found mon.noname-b 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.841+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2483332293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1cc009a20 con 0x7fa1d406e3e0 2023-09-30T00:44:29.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.842+0000 7f69cad9d700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6808/143134 conn(0x7f69ac03b390 0x7f69ac03d7a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.842+0000 7efef9d9b700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6804/143133 conn(0x7efedc03b390 0x7efedc03d7a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.842+0000 7f2af93b1700 1 -- 172.21.15.6:0/63221511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 msgr2=0x7f2af406d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.842+0000 7f2af93b1700 1 --2- 172.21.15.6:0/63221511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 0x7f2af406d580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00ccc0 tx=0x7f2aec005400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec00ab40 con 0x7f2af4135480 2023-09-30T00:44:29.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: dump: 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2afae14700 5 monclient: authenticate success, global_id 4243 2023-09-30T00:44:29.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2aec00acc0 con 0x7f2af4135480 2023-09-30T00:44:29.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== osd.5 v2:172.21.15.22:6804/143133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7efedc03de10 con 0x7efedc03b390 2023-09-30T00:44:29.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== osd.4 v2:172.21.15.22:6808/143134 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f69ac03de10 con 0x7f69ac03b390 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2aec007340 con 0x7f2af4135480 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af0ff9700 10 monclient: dump: 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.916 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.917 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.917 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.917 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7f2af2ffd700 4 set_mon_vals no callback set 2023-09-30T00:44:29.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2483332293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 msgr2=0x7fa1d4139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1d3fff700 1 --2- 172.21.15.6:0/2483332293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d4139b60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1d409f1a0 tx=0x7fa1cc018000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc026430 con 0x7fa1d406e3e0 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: dump: 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.918 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1cc021070 con 0x7fa1d406e3e0 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1cc0267e0 con 0x7fa1d406e3e0 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:29.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1bb7fe700 10 monclient: dump: 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1d1ffb700 4 set_mon_vals no callback set 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 5 monclient: authenticate success, global_id 4246 2023-09-30T00:44:29.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 10 monclient: _renew_subs 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d411e590 con 0x7fa1d406e3e0 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 10 monclient: _renew_subs 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.843+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d413db20 con 0x7fa1d406e3e0 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 10 monclient: _renew_subs 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2af40971c0 con 0x7f2af4135480 2023-09-30T00:44:29.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 10 monclient: _renew_subs 2023-09-30T00:44:29.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2af4098320 con 0x7f2af4135480 2023-09-30T00:44:29.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7fa1da08b700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa1d411e7d0 con 0x7fa1d406e3e0 2023-09-30T00:44:29.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:29.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.844+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2af406edb0 con 0x7f2af4135480 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa1cc023030 con 0x7fa1d406e3e0 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.22:6812/142603 conn(0x7fa1c00357a0 0x7fa1c0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa1cc049c30 con 0x7fa1d406e3e0 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.6:6800/114300 conn(0x7fa1c003b2c0 0x7fa1c003d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 --> v2:172.21.15.6:6800/114300 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa1c003dd40 con 0x7fa1c003b2c0 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1cc04e2a0 con 0x7fa1d406e3e0 2023-09-30T00:44:29.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1bb7fe700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:44:29.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2aec0076f0 con 0x7f2af4135480 2023-09-30T00:44:29.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7f2af0ff9700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.22:6812/142603 conn(0x7f2ad80357a0 0x7f2ad8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.845+0000 7fa1d2ffd700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.6:6800/114300 conn(0x7fa1c003b2c0 0x7fa1c003d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7fa1d2ffd700 10 monclient: get_auth_request con 0x7fa1c003b2c0 auth_method 0 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2aec048880 con 0x7f2af4135480 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7f2af0ff9700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.6:6808/114302 conn(0x7f2ad803b2c0 0x7f2ad803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 --> v2:172.21.15.6:6808/114302 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2ad803dd40 con 0x7f2ad803b2c0 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2aec048b20 con 0x7f2af4135480 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.846+0000 7f2af0ff9700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.847+0000 7fa1d37fe700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.22:6812/142603 conn(0x7fa1c00357a0 0x7fa1c0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.847+0000 7f69c9d9b700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6812/142603 conn(0x7f69ac0357f0 0x7f69ac037ca0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f69cc09fe60 tx=0x7f69c0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.847+0000 7fa1d37fe700 10 monclient: get_auth_request con 0x7fa1c00357a0 auth_method 0 2023-09-30T00:44:29.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.849+0000 7fa1d37fe700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.22:6812/142603 conn(0x7fa1c00357a0 0x7fa1c0037c50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4003a30 tx=0x7fa1c4007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.850+0000 7fa1d2ffd700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.6:6800/114300 conn(0x7fa1c003b2c0 0x7fa1c003d6d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.851+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== osd.1 v2:172.21.15.6:6800/114300 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa1c003dd40 con 0x7fa1c003b2c0 2023-09-30T00:44:29.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.853+0000 7f2af3fff700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.6:6808/114302 conn(0x7f2ad803b2c0 0x7f2ad803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.853+0000 7f2af3fff700 10 monclient: get_auth_request con 0x7f2ad803b2c0 auth_method 0 2023-09-30T00:44:29.927 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:29.928 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.857+0000 7f2af8bb0700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.22:6812/142603 conn(0x7f2ad80357a0 0x7f2ad8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.857+0000 7f2af8bb0700 10 monclient: get_auth_request con 0x7f2ad80357a0 auth_method 0 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.858+0000 7f2af3fff700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.6:6808/114302 conn(0x7f2ad803b2c0 0x7f2ad803d6d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.858+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== osd.2 v2:172.21.15.6:6808/114302 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f2ad803dd40 con 0x7f2ad803b2c0 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.860+0000 7f2af8bb0700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.22:6812/142603 conn(0x7f2ad80357a0 0x7f2ad8037c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0006fd0 tx=0x7f2ae0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.860+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 --> v2:172.21.15.22:6804/143133 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7efefc0692d0 con 0x7efedc03b390 2023-09-30T00:44:29.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7efeea7fc700 1 -- 172.21.15.6:0/123673512 <== osd.5 v2:172.21.15.22:6804/143133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7efefc0692d0 con 0x7efedc03b390 2023-09-30T00:44:29.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6804/143133 conn(0x7efedc03b390 msgr2=0x7efedc03d7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6804/143133 conn(0x7efedc03b390 0x7efedc03d7a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6812/142603 conn(0x7efedc0357f0 msgr2=0x7efedc037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 --2- 172.21.15.6:0/123673512 >> v2:172.21.15.22:6812/142603 conn(0x7efedc0357f0 0x7efedc037ca0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efef00039e0 tx=0x7efef0007ef0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 10 monclient: shutdown 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 msgr2=0x7efefc13ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 --2- 172.21.15.6:0/123673512 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7efefc13b6c0 0x7efefc13ae40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7efef4009eb0 tx=0x7efef400b040 comp rx=0 tx=0).stop 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7efefad9d700 1 -- 172.21.15.6:0/123673512 reap_dead start 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 shutdown_connections 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 >> 172.21.15.6:0/123673512 conn(0x7efefc0a32b0 msgr2=0x7efefc09b5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 shutdown_connections 2023-09-30T00:44:29.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.861+0000 7eff00b13700 1 -- 172.21.15.6:0/123673512 wait complete. 2023-09-30T00:44:29.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.864+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 --> v2:172.21.15.22:6808/143134 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f69cc09ea70 con 0x7f69ac03b390 2023-09-30T00:44:29.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69ba7fc700 1 -- 172.21.15.6:0/2852345781 <== osd.4 v2:172.21.15.22:6808/143134 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f69cc09ea70 con 0x7f69ac03b390 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6808/143134 conn(0x7f69ac03b390 msgr2=0x7f69ac03d7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6808/143134 conn(0x7f69ac03b390 0x7f69ac03d7a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6812/142603 conn(0x7f69ac0357f0 msgr2=0x7f69ac037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 --2- 172.21.15.6:0/2852345781 >> v2:172.21.15.22:6812/142603 conn(0x7f69ac0357f0 0x7f69ac037ca0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f69cc09fe60 tx=0x7f69c0016000 comp rx=0 tx=0).stop 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 10 monclient: shutdown 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 msgr2=0x7f69cc12da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 --2- 172.21.15.6:0/2852345781 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f69cc098670 0x7f69cc12da50 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f69c400bc30 tx=0x7f69c4009840 comp rx=0 tx=0).stop 2023-09-30T00:44:29.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69cad9d700 1 -- 172.21.15.6:0/2852345781 reap_dead start 2023-09-30T00:44:29.934 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 shutdown_connections 2023-09-30T00:44:29.934 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.865+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 >> 172.21.15.6:0/2852345781 conn(0x7f69cc0a5480 msgr2=0x7f69cc0a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.934 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.866+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 shutdown_connections 2023-09-30T00:44:29.934 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.866+0000 7f69d0e0c700 1 -- 172.21.15.6:0/2852345781 wait complete. 2023-09-30T00:44:29.934 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.869+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 --> v2:172.21.15.6:6800/114300 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa1d411d600 con 0x7fa1c003b2c0 2023-09-30T00:44:29.934 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.869+0000 7fa1bb7fe700 1 -- 172.21.15.6:0/2483332293 <== osd.1 v2:172.21.15.6:6800/114300 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa1d411d600 con 0x7fa1c003b2c0 2023-09-30T00:44:29.935 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 >> v2:172.21.15.6:6800/114300 conn(0x7fa1c003b2c0 msgr2=0x7fa1c003d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.935 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.6:6800/114300 conn(0x7fa1c003b2c0 0x7fa1c003d6d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.935 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 >> v2:172.21.15.22:6812/142603 conn(0x7fa1c00357a0 msgr2=0x7fa1c0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 --2- 172.21.15.6:0/2483332293 >> v2:172.21.15.22:6812/142603 conn(0x7fa1c00357a0 0x7fa1c0037c50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4003a30 tx=0x7fa1c4007ea0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 10 monclient: shutdown 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 msgr2=0x7fa1d4139b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 --2- 172.21.15.6:0/2483332293 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa1d406e3e0 0x7fa1d4139b60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa1d409f1a0 tx=0x7fa1cc018000 comp rx=0 tx=0).stop 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1d3fff700 1 -- 172.21.15.6:0/2483332293 reap_dead start 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 shutdown_connections 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 >> 172.21.15.6:0/2483332293 conn(0x7fa1d40680c0 msgr2=0x7fa1d409f1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 shutdown_connections 2023-09-30T00:44:29.936 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.870+0000 7fa1da08b700 1 -- 172.21.15.6:0/2483332293 wait complete. 2023-09-30T00:44:29.937 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:29.937 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:44:29.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.876+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 --> v2:172.21.15.6:6808/114302 -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2af406dd00 con 0x7f2ad803b2c0 2023-09-30T00:44:29.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2af0ff9700 1 -- 172.21.15.6:0/63221511 <== osd.2 v2:172.21.15.6:6808/114302 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f2af406dd00 con 0x7f2ad803b2c0 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 >> v2:172.21.15.6:6808/114302 conn(0x7f2ad803b2c0 msgr2=0x7f2ad803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.6:6808/114302 conn(0x7f2ad803b2c0 0x7f2ad803d6d0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 >> v2:172.21.15.22:6812/142603 conn(0x7f2ad80357a0 msgr2=0x7f2ad8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 --2- 172.21.15.6:0/63221511 >> v2:172.21.15.22:6812/142603 conn(0x7f2ad80357a0 0x7f2ad8037c50 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f2ae0006fd0 tx=0x7f2ae0008040 comp rx=0 tx=0).stop 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 10 monclient: shutdown 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 msgr2=0x7f2af406d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 --2- 172.21.15.6:0/63221511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2af4135480 0x7f2af406d580 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f2aec00ccc0 tx=0x7f2aec005400 comp rx=0 tx=0).stop 2023-09-30T00:44:29.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2af93b1700 1 -- 172.21.15.6:0/63221511 reap_dead start 2023-09-30T00:44:29.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 shutdown_connections 2023-09-30T00:44:29.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 >> 172.21.15.6:0/63221511 conn(0x7f2af40a3320 msgr2=0x7f2af4069490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:29.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 shutdown_connections 2023-09-30T00:44:29.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:29.877+0000 7f2afae14700 1 -- 172.21.15.6:0/63221511 wait complete. 2023-09-30T00:44:29.940 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:29.941 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:44:29.941 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:29.942 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:44:29.942 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2023-09-30T00:44:29.943 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:44:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.131+0000 7f57eb022700 10 monclient: get_monmap_and_config 2023-09-30T00:44:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.131+0000 7f57eb022700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.131+0000 7f57eb022700 10 monclient: monmap: 2023-09-30T00:44:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:30.132093+0000 2023-09-30T00:44:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:30.132093+0000 2023-09-30T00:44:30.133 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:30.134 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:30.135 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.133+0000 7f57eb022700 1 Processor -- start 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.133+0000 7f57eb022700 1 -- start start 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.133+0000 7f57eb022700 10 monclient: init 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f57e4098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f57e4132d00 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f57e4138d40 0x7f57e413b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient(hunting): picked mon.noname-c con 0x7f57e4138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:30.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40a1000 con 0x7f57e4098670 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57e40a0730 con 0x7f57e4132d00 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57e4099150 con 0x7f57e4138d40 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57eb022700 10 monclient(hunting): authenticate will time out at 1266.431166s 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f57e4138d40 0x7f57e413b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f57e4138d40 0x7f57e413b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.6:49294/0 (socket says 172.21.15.6:49294) 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 -- 172.21.15.6:0/2524626887 learned_addr learned my addr 172.21.15.6:0/2524626887 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): get_auth_request con 0x7f57e4138d40 auth_method 0 2023-09-30T00:44:30.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient(hunting): handle_auth_done global_id 4249 payload 274 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 -- 172.21.15.6:0/2524626887 >> v1:172.21.15.22:6789/0 conn(0x7f57e4132d00 legacy=0x7f57e4137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 -- 172.21.15.6:0/2524626887 >> v1:172.21.15.6:6789/0 conn(0x7f57e4098670 legacy=0x7f57e4098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 monclient: found mon.noname-c 2023-09-30T00:44:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.134+0000 7f57e95bf700 1 -- 172.21.15.6:0/2524626887 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57e4068df0 con 0x7f57e4138d40 2023-09-30T00:44:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.137+0000 7f57e95bf700 1 -- 172.21.15.6:0/2524626887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4138d40 msgr2=0x7f57e413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.137+0000 7f57e95bf700 1 --2- 172.21.15.6:0/2524626887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4138d40 0x7f57e413b120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009f20 tx=0x7f57dc009840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e99f69761b191ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 1 -- 172.21.15.6:0/2524626887 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00e070 con 0x7f57e4138d40 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: dump: 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.141 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 1 -- 172.21.15.6:0/2524626887 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57dc00bc50 con 0x7f57e4138d40 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 1 -- 172.21.15.6:0/2524626887 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57dc012430 con 0x7f57e4138d40 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57db7fe700 10 monclient: dump: 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.143 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57eb022700 5 monclient: authenticate success, global_id 4249 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57eb022700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:30.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.139+0000 7f57eb022700 4 set_mon_vals no callback set 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 10 monclient: shutdown 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 -- 172.21.15.6:0/2524626887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4138d40 msgr2=0x7f57e413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 --2- 172.21.15.6:0/2524626887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4138d40 0x7f57e413b120 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009f20 tx=0x7f57dc009840 comp rx=0 tx=0).stop 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 -- 172.21.15.6:0/2524626887 shutdown_connections 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 --2- 172.21.15.6:0/2524626887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4138d40 0x7f57e413b120 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 -- 172.21.15.6:0/2524626887 >> 172.21.15.6:0/2524626887 conn(0x7f57e40a5480 msgr2=0x7f57e40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 -- 172.21.15.6:0/2524626887 shutdown_connections 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.140+0000 7f57eb022700 1 -- 172.21.15.6:0/2524626887 wait complete. 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.141+0000 7f57eb022700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.141+0000 7f57eb022700 10 monclient: monmap: 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:30.146 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.142+0000 7f786b6ec700 10 monclient: get_monmap_and_config 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.142+0000 7f786b6ec700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.142+0000 7f786b6ec700 10 monclient: monmap: 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:30.142471+0000 2023-09-30T00:44:30.147 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:30.142471+0000 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:30.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.141+0000 7f57eb022700 1 Processor -- start 2023-09-30T00:44:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.142+0000 7f57eb022700 1 -- start start 2023-09-30T00:44:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.142+0000 7f57eb022700 10 monclient: init 2023-09-30T00:44:30.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f786b6ec700 1 Processor -- start 2023-09-30T00:44:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:44:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57e4098670 0x7f57e412f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): picked mon.noname-a con 0x7f57e4098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 0x7f57e4099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): picked mon.noname-b con 0x7f57e4132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57e4138d40 0x7f57e40997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): picked mon.noname-c con 0x7f57e4138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e40a1500 con 0x7f57e4098670 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e4130390 con 0x7f57e4132d00 2023-09-30T00:44:30.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57e409bec0 con 0x7f57e4138d40 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57eb022700 10 monclient(hunting): authenticate will time out at 1266.440692s 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.143+0000 7f57e8dbe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57e4098670 0x7f57e412f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57e4098670 0x7f57e412f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41986/0 (socket says 172.21.15.6:41986) 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 1 -- 172.21.15.6:0/2281100517 learned_addr learned my addr 172.21.15.6:0/2281100517 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): get_auth_request con 0x7f57e4098670 auth_method 0 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 0x7f57e4099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): get_auth_request con 0x7f57e4132d00 auth_method 0 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57e8dbe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient(hunting): handle_auth_done global_id 4252 payload 824 2023-09-30T00:44:30.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 -- 172.21.15.6:0/2281100517 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57e4138d40 msgr2=0x7f57e40997d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57e4138d40 0x7f57e40997d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 -- 172.21.15.6:0/2281100517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57e4098670 msgr2=0x7f57e412f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57e4098670 0x7f57e412f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 monclient: found mon.noname-b 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.144+0000 7f57dbfff700 1 -- 172.21.15.6:0/2281100517 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57dc009ae0 con 0x7f57e4132d00 2023-09-30T00:44:30.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 1 -- start start 2023-09-30T00:44:30.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient: init 2023-09-30T00:44:30.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T00:44:30.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f7864098670 0x7f7864132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): picked mon.noname-c con 0x7f7864098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:30.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f7864098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:30.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7864098e90 0x7f786412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): picked mon.noname-a con 0x7f7864098e90 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:30.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f78640a1000 con 0x7f7864098a40 2023-09-30T00:44:30.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78640a0a50 con 0x7f7864098e90 2023-09-30T00:44:30.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f78640a0730 con 0x7f7864098670 2023-09-30T00:44:30.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.145+0000 7f786b6ec700 10 monclient(hunting): authenticate will time out at 1266.442950s 2023-09-30T00:44:30.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7864098e90 0x7f786412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7864098e90 0x7f786412f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42002/0 (socket says 172.21.15.6:42002) 2023-09-30T00:44:30.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 1 -- 172.21.15.6:0/3615149425 learned_addr learned my addr 172.21.15.6:0/3615149425 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): get_auth_request con 0x7f7864098e90 auth_method 0 2023-09-30T00:44:30.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 1 --2- 172.21.15.6:0/3615149425 >> v2:172.21.15.22:3300/0 conn(0x7f7864098670 0x7f7864132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): get_auth_request con 0x7f7864098670 auth_method 0 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869c89700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient(hunting): handle_auth_done global_id 4255 payload 274 2023-09-30T00:44:30.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 1 -- 172.21.15.6:0/3615149425 >> v2:172.21.15.6:3300/0 conn(0x7f7864098e90 msgr2=0x7f786412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 1 --2- 172.21.15.6:0/3615149425 >> v2:172.21.15.6:3300/0 conn(0x7f7864098e90 0x7f786412f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 1 -- 172.21.15.6:0/3615149425 >> v1:172.21.15.22:6789/0 conn(0x7f7864098a40 legacy=0x7f786412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 1 monclient: found mon.noname-c 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f7869488700 1 -- 172.21.15.6:0/3615149425 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7864068df0 con 0x7f7864098670 2023-09-30T00:44:30.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f57dbfff700 1 -- 172.21.15.6:0/2281100517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 msgr2=0x7f57e4099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.146+0000 7f57dbfff700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 0x7f57e4099290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c4e0 tx=0x7f57e0007ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e00117a0 con 0x7f57e4132d00 2023-09-30T00:44:30.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: dump: 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.167 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 5 monclient: authenticate success, global_id 4252 2023-09-30T00:44:30.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57e0011920 con 0x7f57e4132d00 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57e0011ce0 con 0x7f57e4132d00 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57d8ff9700 10 monclient: dump: 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.169 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.170 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57daffd700 4 set_mon_vals no callback set 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 10 monclient: _renew_subs 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e409c1a0 con 0x7f57e4132d00 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 10 monclient: _renew_subs 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.147+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e409b680 con 0x7f57e4132d00 2023-09-30T00:44:30.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.148+0000 7f57eb022700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.148+0000 7f57eb022700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.148+0000 7f57eb022700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.148+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e409bd00 con 0x7f57e4132d00 2023-09-30T00:44:30.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.148+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f57e0012630 con 0x7f57e4132d00 2023-09-30T00:44:30.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.148+0000 7f57d8ff9700 1 --2- 172.21.15.6:0/2281100517 >> v2:172.21.15.22:6812/142603 conn(0x7f57d00357d0 0x7f57d0037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.149+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f57e0047570 con 0x7f57e4132d00 2023-09-30T00:44:30.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.149+0000 7f7869488700 1 -- 172.21.15.6:0/3615149425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 msgr2=0x7f7864132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.149+0000 7f7869488700 1 --2- 172.21.15.6:0/3615149425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f7864132d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f786409fe60 tx=0x7f78600091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3dd57c341e8354b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.150+0000 7f57e8dbe700 1 --2- 172.21.15.6:0/2281100517 >> v2:172.21.15.22:6812/142603 conn(0x7f57d00357d0 0x7f57d0037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.150+0000 7f57e8dbe700 10 monclient: get_auth_request con 0x7f57d00357d0 auth_method 0 2023-09-30T00:44:30.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.152+0000 7f57e8dbe700 1 --2- 172.21.15.6:0/2281100517 >> v2:172.21.15.22:6812/142603 conn(0x7f57d00357d0 0x7f57d0037c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009b80 tx=0x7f57d40091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.152+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 DispatchQueue::entry inject delay of 0.002000 2023-09-30T00:44:30.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 1 -- 172.21.15.6:0/3615149425 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860011840 con 0x7f7864098670 2023-09-30T00:44:30.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: dump: 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.175 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 1 -- 172.21.15.6:0/3615149425 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f78600119a0 con 0x7f7864098670 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 1 -- 172.21.15.6:0/3615149425 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7860011d40 con 0x7f7864098670 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f785bfff700 10 monclient: dump: 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.177 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f786b6ec700 5 monclient: authenticate success, global_id 4255 2023-09-30T00:44:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f786b6ec700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.154+0000 7f786b6ec700 4 set_mon_vals no callback set 2023-09-30T00:44:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57e0002090 con 0x7f57e4132d00 2023-09-30T00:44:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f57d8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f57d8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 10 monclient: shutdown 2023-09-30T00:44:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 1 -- 172.21.15.6:0/3615149425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 msgr2=0x7f7864132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3615149425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f7864132d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f786409fe60 tx=0x7f78600091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 1 -- 172.21.15.6:0/3615149425 shutdown_connections 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3615149425 >> v2:172.21.15.6:3300/0 conn(0x7f7864098e90 0x7f786412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3615149425 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f7864132d00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.155+0000 7f786b6ec700 1 -- 172.21.15.6:0/3615149425 >> 172.21.15.6:0/3615149425 conn(0x7f78640a5480 msgr2=0x7f78640a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.156+0000 7f786b6ec700 1 -- 172.21.15.6:0/3615149425 shutdown_connections 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.156+0000 7f786b6ec700 1 -- 172.21.15.6:0/3615149425 wait complete. 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.156+0000 7f786b6ec700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.156+0000 7f786b6ec700 10 monclient: monmap: 2023-09-30T00:44:30.184 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.185 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:30.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.157+0000 7f786b6ec700 1 Processor -- start 2023-09-30T00:44:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.157+0000 7f786b6ec700 1 -- start start 2023-09-30T00:44:30.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.157+0000 7f786b6ec700 10 monclient: init 2023-09-30T00:44:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f786412c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): picked mon.noname-b con 0x7f7864098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:30.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7864098a40 0x7f786409cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): picked mon.noname-a con 0x7f7864098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 0x7f786409d2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): picked mon.noname-c con 0x7f7864098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786411c5c0 con 0x7f7864098a40 2023-09-30T00:44:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f786411cbf0 con 0x7f7864098670 2023-09-30T00:44:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f786409f330 con 0x7f7864098e90 2023-09-30T00:44:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f786b6ec700 10 monclient(hunting): authenticate will time out at 1266.455639s 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 0x7f786409d2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 0x7f786409d2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36542/0 (socket says 172.21.15.6:36542) 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 1 -- 172.21.15.6:0/3707059882 learned_addr learned my addr 172.21.15.6:0/3707059882 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 10 monclient(hunting): get_auth_request con 0x7f7864098e90 auth_method 0 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.158+0000 7f7869c89700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7868c87700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7864098a40 0x7f786409cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7868c87700 10 monclient(hunting): get_auth_request con 0x7f7864098a40 auth_method 0 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7868c87700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7868c87700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7868c87700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f786412c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): get_auth_request con 0x7f7864098670 auth_method 0 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869488700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 10 monclient(hunting): handle_auth_done global_id 4256 payload 824 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 1 -- 172.21.15.6:0/3707059882 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 msgr2=0x7f786412c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f786412c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 1 -- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7864098a40 msgr2=0x7f786409cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7864098a40 0x7f786409cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 1 monclient: found mon.noname-c 2023-09-30T00:44:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.159+0000 7f7869c89700 1 -- 172.21.15.6:0/3707059882 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7860009420 con 0x7f7864098e90 2023-09-30T00:44:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.161+0000 7f7869c89700 1 -- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 msgr2=0x7f786409d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.161+0000 7f7869c89700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 0x7f786409d2e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f785c009e50 tx=0x7f785c00b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.161+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c009c30 con 0x7f7864098e90 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.161+0000 7f78597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.161+0000 7f78597fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.161+0000 7f78597fa700 10 monclient: dump: 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.195 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f785c00f040 con 0x7f7864098e90 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f785c00cda0 con 0x7f7864098e90 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f78597fa700 10 monclient: dump: 2023-09-30T00:44:30.197 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.198 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f785b7fe700 4 set_mon_vals no callback set 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 5 monclient: authenticate success, global_id 4256 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 10 monclient: _renew_subs 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7864131300 con 0x7f7864098e90 2023-09-30T00:44:30.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 10 monclient: _renew_subs 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.162+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f786409e8c0 con 0x7f7864098e90 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f785c014650 con 0x7f7864098e90 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f78597fa700 1 --2- 172.21.15.6:0/3707059882 >> v2:172.21.15.22:6812/142603 conn(0x7f78480357a0 0x7f7848037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f785c047fe0 con 0x7f7864098e90 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f786b6ec700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f786b6ec700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f786b6ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7864098120 con 0x7f7864098e90 2023-09-30T00:44:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f7869488700 1 --2- 172.21.15.6:0/3707059882 >> v2:172.21.15.22:6812/142603 conn(0x7f78480357a0 0x7f7848037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.164+0000 7f7869488700 10 monclient: get_auth_request con 0x7f78480357a0 auth_method 0 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.168+0000 7fc91b652700 10 monclient: get_monmap_and_config 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.168+0000 7fc91b652700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.168+0000 7fc91b652700 10 monclient: monmap: 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:30.168435+0000 2023-09-30T00:44:30.202 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:30.168435+0000 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:30.203 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.172+0000 7fc91b652700 1 Processor -- start 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.172+0000 7fc91b652700 1 -- start start 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.172+0000 7fc91b652700 10 monclient: init 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient: _add_conns ranks=[2,0,3,4,1] 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 0x7fc914069730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): picked mon.noname-b con 0x7fc91409cd40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc91409a8f0 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc91413b6c0 0x7fc91413daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): picked mon.noname-a con 0x7fc91413b6c0 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc914068820 con 0x7fc91409a8f0 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91406a0b0 con 0x7fc91413b6c0 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91409f090 con 0x7fc91409cd40 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc91b652700 10 monclient(hunting): authenticate will time out at 1266.470414s 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc9193ee700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 0x7fc914069730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.173+0000 7fc9193ee700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 0x7fc914069730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36544/0 (socket says 172.21.15.6:36544) 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 1 -- 172.21.15.6:0/1805799527 learned_addr learned my addr 172.21.15.6:0/1805799527 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): get_auth_request con 0x7fc91409cd40 auth_method 0 2023-09-30T00:44:30.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.167+0000 7f7869488700 1 --2- 172.21.15.6:0/3707059882 >> v2:172.21.15.22:6812/142603 conn(0x7f78480357a0 0x7f7848037c50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f78600093a0 tx=0x7f7860016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.172+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f785c01d400 con 0x7f7864098e90 2023-09-30T00:44:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.172+0000 7f78597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.172+0000 7f78597fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:30.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient(hunting): handle_auth_done global_id 4259 payload 274 2023-09-30T00:44:30.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 1 -- 172.21.15.6:0/1805799527 >> v2:172.21.15.6:3300/0 conn(0x7fc91413b6c0 msgr2=0x7fc91413daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 1 --2- 172.21.15.6:0/1805799527 >> v2:172.21.15.6:3300/0 conn(0x7fc91413b6c0 0x7fc91413daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 1 -- 172.21.15.6:0/1805799527 >> v1:172.21.15.6:6789/0 conn(0x7fc91409a8f0 legacy=0x7fc91409d110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 1 monclient: found mon.noname-b 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.174+0000 7fc9193ee700 1 -- 172.21.15.6:0/1805799527 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91406a6e0 con 0x7fc91409cd40 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.179+0000 7fc9193ee700 1 -- 172.21.15.6:0/1805799527 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 msgr2=0x7fc914069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.179+0000 7fc9193ee700 1 --2- 172.21.15.6:0/1805799527 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 0x7fc914069730 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc904009990 tx=0x7fc904009220 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e13feec6d3d238e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 1 -- 172.21.15.6:0/1805799527 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904005e20 con 0x7fc91409cd40 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: dump: 2023-09-30T00:44:30.211 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.212 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.213 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 1 -- 172.21.15.6:0/1805799527 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90400b610 con 0x7fc91409cd40 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 1 -- 172.21.15.6:0/1805799527 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90400b9b0 con 0x7fc91409cd40 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc90bfff700 10 monclient: dump: 2023-09-30T00:44:30.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.215 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 5 monclient: authenticate success, global_id 4259 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 4 set_mon_vals no callback set 2023-09-30T00:44:30.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 10 monclient: shutdown 2023-09-30T00:44:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 1 -- 172.21.15.6:0/1805799527 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 msgr2=0x7fc914069730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.180+0000 7fc91b652700 1 --2- 172.21.15.6:0/1805799527 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 0x7fc914069730 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc904009990 tx=0x7fc904009220 comp rx=0 tx=0).stop 2023-09-30T00:44:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7fc91b652700 1 -- 172.21.15.6:0/1805799527 shutdown_connections 2023-09-30T00:44:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7fc91b652700 1 --2- 172.21.15.6:0/1805799527 >> v2:172.21.15.6:3300/0 conn(0x7fc91413b6c0 0x7fc91413daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7fc91b652700 1 --2- 172.21.15.6:0/1805799527 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409cd40 0x7fc914069730 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7fc91b652700 1 -- 172.21.15.6:0/1805799527 >> 172.21.15.6:0/1805799527 conn(0x7fc9140a32b0 msgr2=0x7fc9140a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7f0a07b68700 10 monclient: get_monmap_and_config 2023-09-30T00:44:30.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7f0a07b68700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.181+0000 7f0a07b68700 10 monclient: monmap: 2023-09-30T00:44:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:30.182019+0000 2023-09-30T00:44:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:30.182019+0000 2023-09-30T00:44:30.219 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:30.220 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7fc91b652700 1 -- 172.21.15.6:0/1805799527 shutdown_connections 2023-09-30T00:44:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7fc91b652700 1 -- 172.21.15.6:0/1805799527 wait complete. 2023-09-30T00:44:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7fc91b652700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7fc91b652700 10 monclient: monmap: 2023-09-30T00:44:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.221 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 Processor -- start 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 -- start start 2023-09-30T00:44:30.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient: init 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7ff4233eb700 10 monclient: get_monmap_and_config 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7ff4233eb700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.182+0000 7ff4233eb700 10 monclient: monmap: 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:30.182674+0000 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:30.182674+0000 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:30.223 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:30.224 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:30.224 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:30.224 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:30.224 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:30.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 0x7fc91413b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): picked mon.noname-c con 0x7fc91409a8f0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc91409cd40 0x7fc9140977a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): picked mon.noname-b con 0x7fc91409cd40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc91413b6c0 0x7fc914097ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): picked mon.noname-a con 0x7fc91413b6c0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91406f1d0 con 0x7fc91413b6c0 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91409ae50 con 0x7fc91409cd40 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91411acc0 con 0x7fc91409a8f0 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.183+0000 7fc91b652700 10 monclient(hunting): authenticate will time out at 1266.480977s 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 0x7fc91413b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 0x7fc91413b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36548/0 (socket says 172.21.15.6:36548) 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 -- 172.21.15.6:0/264974701 learned_addr learned my addr 172.21.15.6:0/264974701 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): get_auth_request con 0x7fc91409a8f0 auth_method 0 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc91413b6c0 0x7fc914097ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): get_auth_request con 0x7fc91413b6c0 auth_method 0 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc91409cd40 0x7fc9140977a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): get_auth_request con 0x7fc91409cd40 auth_method 0 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc919bef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient(hunting): handle_auth_done global_id 4262 payload 824 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 -- 172.21.15.6:0/264974701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc91409cd40 msgr2=0x7fc9140977a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc91409cd40 0x7fc9140977a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 -- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc91413b6c0 msgr2=0x7fc914097ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc91413b6c0 0x7fc914097ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 monclient: found mon.noname-c 2023-09-30T00:44:30.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc9193ee700 1 -- 172.21.15.6:0/264974701 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc904009480 con 0x7fc91409a8f0 2023-09-30T00:44:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7fc918bed700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc91409cd40 0x7fc9140977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.184+0000 7f0a07b68700 1 Processor -- start 2023-09-30T00:44:30.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7ff4233eb700 1 Processor -- start 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 1 -- start start 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient: init 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0a00098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): picked mon.noname-b con 0x7f0a00132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0a00137b60 0x7f0a00139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): picked mon.noname-a con 0x7f0a00137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000a1000 con 0x7f0a00098670 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a00099150 con 0x7f0a00137b60 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000a0a50 con 0x7f0a00132d00 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7f0a07b68700 10 monclient(hunting): authenticate will time out at 1266.482959s 2023-09-30T00:44:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7ff4233eb700 1 -- start start 2023-09-30T00:44:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.185+0000 7ff4233eb700 10 monclient: init 2023-09-30T00:44:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient: _add_conns ranks=[2,1,4,3,0] 2023-09-30T00:44:30.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 0x7ff41c097610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): picked mon.noname-b con 0x7ff41c09baf0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff41c09bec0 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7ff41c133800 0x7ff41c135be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): picked mon.noname-c con 0x7ff41c133800 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff41c0964a0 con 0x7ff41c09bec0 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff41c09c9a0 con 0x7ff41c133800 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c09d860 con 0x7ff41c09baf0 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff4233eb700 10 monclient(hunting): authenticate will time out at 1266.483294s 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a05904700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0a00098670 0x7f0a00098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60874/0 (socket says 172.21.15.6:60874) 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a05904700 1 -- 172.21.15.6:0/562712365 learned_addr learned my addr 172.21.15.6:0/562712365 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 1 --2- 172.21.15.6:0/562712365 >> v2:172.21.15.6:3300/0 conn(0x7f0a00137b60 0x7f0a00139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): get_auth_request con 0x7f0a00137b60 auth_method 0 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff420986700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7ff41c09bec0 0x7ff41c09c290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57352/0 (socket says 172.21.15.6:57352) 2023-09-30T00:44:30.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff420986700 1 -- 172.21.15.6:0/2303308613 learned_addr learned my addr 172.21.15.6:0/2303308613 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:30.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff421988700 1 --2- 172.21.15.6:0/2303308613 >> v2:172.21.15.22:3300/0 conn(0x7ff41c133800 0x7ff41c135be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7f0a06105700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff421988700 10 monclient(hunting): get_auth_request con 0x7ff41c133800 auth_method 0 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff421988700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff421988700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.186+0000 7ff421988700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421988700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421988700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421988700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f6e867bf700 10 monclient: get_monmap_and_config 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f6e867bf700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f6e867bf700 10 monclient: monmap: 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:30.187382+0000 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:30.187382+0000 2023-09-30T00:44:30.238 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.239 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 10 monclient(hunting): handle_auth_done global_id 4257 payload 274 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 1 -- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 msgr2=0x7f0a00137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 1 --2- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 1 -- 172.21.15.6:0/562712365 >> v1:172.21.15.6:6789/0 conn(0x7f0a00098670 legacy=0x7f0a00098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 1 monclient: found mon.noname-a 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7f0a06105700 1 -- 172.21.15.6:0/562712365 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000a0730 con 0x7f0a00137b60 2023-09-30T00:44:30.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 1 --2- 172.21.15.6:0/2303308613 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 0x7ff41c097610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): get_auth_request con 0x7ff41c09baf0 auth_method 0 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9193ee700 1 -- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 msgr2=0x7fc91413b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9193ee700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 0x7fc91413b0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc90400b3c0 tx=0x7fc904015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc904011a80 con 0x7fc91409a8f0 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: dump: 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.242 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient(hunting): handle_auth_done global_id 4265 payload 274 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 1 -- 172.21.15.6:0/2303308613 >> v2:172.21.15.22:3300/0 conn(0x7ff41c133800 msgr2=0x7ff41c135be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 1 --2- 172.21.15.6:0/2303308613 >> v2:172.21.15.22:3300/0 conn(0x7ff41c133800 0x7ff41c135be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 1 -- 172.21.15.6:0/2303308613 >> v1:172.21.15.22:6789/0 conn(0x7ff41c09bec0 legacy=0x7ff41c09c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 1 monclient: found mon.noname-b 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7ff421187700 1 -- 172.21.15.6:0/2303308613 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff41c0961c0 con 0x7ff41c09baf0 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc904011c00 con 0x7fc91409a8f0 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc90401d430 con 0x7fc91409a8f0 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:30.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.187+0000 7fc9097fa700 10 monclient: dump: 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 5 monclient: authenticate success, global_id 4262 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 10 monclient: _renew_subs 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc90b7fe700 4 set_mon_vals no callback set 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9140993d0 con 0x7fc91409a8f0 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 10 monclient: _renew_subs 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc914119dd0 con 0x7fc91409a8f0 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.188+0000 7fc91b652700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.189+0000 7fc91b652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.189+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc914072b00 con 0x7fc91409a8f0 2023-09-30T00:44:30.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.189+0000 7f6e867bf700 1 Processor -- start 2023-09-30T00:44:30.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.189+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc90401d7e0 con 0x7fc91409a8f0 2023-09-30T00:44:30.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.189+0000 7f6e867bf700 1 -- start start 2023-09-30T00:44:30.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.189+0000 7f6e867bf700 10 monclient: init 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7fc9097fa700 1 --2- 172.21.15.6:0/264974701 >> v2:172.21.15.22:6812/142603 conn(0x7fc8f80357f0 0x7fc8f8037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc904049a90 con 0x7fc91409a8f0 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a06105700 1 -- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 msgr2=0x7f0a00139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a06105700 1 --2- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00139f40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009910 tx=0x7f09fc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b45ea69292a1ca0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 1 -- 172.21.15.6:0/562712365 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc00b9f0 con 0x7f0a00137b60 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: dump: 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a07b68700 5 monclient: authenticate success, global_id 4257 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a07b68700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 1 -- 172.21.15.6:0/562712365 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc00bb50 con 0x7f0a00137b60 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 1 -- 172.21.15.6:0/562712365 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc011420 con 0x7f0a00137b60 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a04902700 10 monclient: dump: 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.251 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a07b68700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f0a07b68700 4 set_mon_vals no callback set 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 10 monclient: shutdown 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 1 -- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 msgr2=0x7f0a00139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 1 --2- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00139f40 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f09fc009910 tx=0x7f09fc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f6e867bf700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f6e867bf700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f6e867bf700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e8009c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f6e867bf700 10 monclient(hunting): picked mon.noname-b con 0x7f6e8009b350 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7f6e867bf700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6e8009dc20 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6e8009e740 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8009f120 con 0x7f6e8009e740 2023-09-30T00:44:30.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e8009bd00 con 0x7f6e8009dc20 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8009c2c0 con 0x7f6e8009b350 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e867bf700 10 monclient(hunting): authenticate will time out at 1266.488174s 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e7f7fe700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f6e8009dc20 0x7f6e8009e030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57356/0 (socket says 172.21.15.6:57356) 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f6e7f7fe700 1 -- 172.21.15.6:0/2911224515 learned_addr learned my addr 172.21.15.6:0/2911224515 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7fc918bed700 1 --2- 172.21.15.6:0/264974701 >> v2:172.21.15.22:6812/142603 conn(0x7fc8f80357f0 0x7fc8f8037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.190+0000 7fc918bed700 10 monclient: get_auth_request con 0x7fc8f80357f0 auth_method 0 2023-09-30T00:44:30.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 1 -- 172.21.15.6:0/562712365 shutdown_connections 2023-09-30T00:44:30.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 1 --2- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00139f40 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 1 --2- 172.21.15.6:0/562712365 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.191+0000 7f0a07b68700 1 -- 172.21.15.6:0/562712365 >> 172.21.15.6:0/562712365 conn(0x7f0a000a5480 msgr2=0x7f0a000a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 1 --2- 172.21.15.6:0/2911224515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e8009c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): get_auth_request con 0x7f6e8009b350 auth_method 0 2023-09-30T00:44:30.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7fc918bed700 1 --2- 172.21.15.6:0/264974701 >> v2:172.21.15.22:6812/142603 conn(0x7fc8f80357f0 0x7fc8f8037ca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc914098fd0 tx=0x7fc910009060 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7f7fe700 1 -- 172.21.15.6:0/2911224515 >> v1:172.21.15.22:6789/0 conn(0x7f6e8009dc20 legacy=0x7f6e8009e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e84d5c700 1 -- 172.21.15.6:0/2911224515 >> v1:172.21.15.6:6789/0 conn(0x7f6e8009e740 legacy=0x7f6e80137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_done global_id 4268 payload 274 2023-09-30T00:44:30.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 1 -- 172.21.15.6:0/2911224515 >> v1:172.21.15.22:6789/0 conn(0x7f6e8009dc20 legacy=0x7f6e8009e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 1 -- 172.21.15.6:0/2911224515 >> v1:172.21.15.6:6789/0 conn(0x7f6e8009e740 legacy=0x7f6e80137b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 1 monclient: found mon.noname-b 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f6e7ffff700 1 -- 172.21.15.6:0/2911224515 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e8009b720 con 0x7f6e8009b350 2023-09-30T00:44:30.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f0a07b68700 1 -- 172.21.15.6:0/562712365 shutdown_connections 2023-09-30T00:44:30.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.193+0000 7f0a07b68700 1 -- 172.21.15.6:0/562712365 wait complete. 2023-09-30T00:44:30.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7ff421187700 1 -- 172.21.15.6:0/2303308613 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 msgr2=0x7ff41c097610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7ff421187700 1 --2- 172.21.15.6:0/2303308613 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 0x7ff41c097610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0098d0 tx=0x7ff40c009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e59fa02d0f3353f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7f0a07b68700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7f0a07b68700 10 monclient: monmap: 2023-09-30T00:44:30.260 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.260 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:30.260 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7f0a07b68700 1 Processor -- start 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7f0a07b68700 1 -- start start 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.194+0000 7f0a07b68700 10 monclient: init 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a00098670 0x7f0a0009edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): picked mon.noname-b con 0x7f0a00098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00141ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): picked mon.noname-c con 0x7f0a00132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00136aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): picked mon.noname-a con 0x7f0a00137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a000a1500 con 0x7f0a00137b60 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0009bdd0 con 0x7f0a00098670 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a0009c0b0 con 0x7f0a00132d00 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a07b68700 10 monclient(hunting): authenticate will time out at 1266.492315s 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00136aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a00098670 0x7f0a0009edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00136aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42044/0 (socket says 172.21.15.6:42044) 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 -- 172.21.15.6:0/2224975489 learned_addr learned my addr 172.21.15.6:0/2224975489 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): get_auth_request con 0x7f0a00137b60 auth_method 0 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): get_auth_request con 0x7f0a00098670 auth_method 0 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05103700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00141ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05103700 10 monclient(hunting): get_auth_request con 0x7f0a00132d00 auth_method 0 2023-09-30T00:44:30.264 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05103700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05103700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05103700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient(hunting): handle_auth_done global_id 4263 payload 824 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 -- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 msgr2=0x7f0a00141ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00141ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 -- 172.21.15.6:0/2224975489 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a00098670 msgr2=0x7f0a0009edc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a00098670 0x7f0a0009edc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 monclient: found mon.noname-a 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.265 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a06105700 1 -- 172.21.15.6:0/2224975489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09fc009420 con 0x7f0a00137b60 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.195+0000 7f0a05904700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a00098670 0x7f0a0009edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 1 -- 172.21.15.6:0/2303308613 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c005e20 con 0x7ff41c09baf0 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: dump: 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.266 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 1 -- 172.21.15.6:0/2303308613 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff40c006030 con 0x7ff41c09baf0 2023-09-30T00:44:30.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc90401de60 con 0x7fc91409a8f0 2023-09-30T00:44:30.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7fc9097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7fc9097fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:30.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7f6e7ffff700 1 -- 172.21.15.6:0/2911224515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 msgr2=0x7f6e8009c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7f6e7ffff700 1 --2- 172.21.15.6:0/2911224515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e8009c9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800b050 tx=0x7f6e78009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0029d22172dcb7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 1 -- 172.21.15.6:0/2303308613 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff40c0063b0 con 0x7ff41c09baf0 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff413fff700 10 monclient: dump: 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.269 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff4233eb700 5 monclient: authenticate success, global_id 4265 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff4233eb700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.196+0000 7ff4233eb700 4 set_mon_vals no callback set 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 10 monclient: shutdown 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 -- 172.21.15.6:0/2303308613 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 msgr2=0x7ff41c097610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 --2- 172.21.15.6:0/2303308613 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 0x7ff41c097610 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7ff40c0098d0 tx=0x7ff40c009120 comp rx=0 tx=0).stop 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 -- 172.21.15.6:0/2303308613 shutdown_connections 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 --2- 172.21.15.6:0/2303308613 >> v2:172.21.15.22:3300/0 conn(0x7ff41c133800 0x7ff41c135be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 --2- 172.21.15.6:0/2303308613 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09baf0 0x7ff41c097610 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 -- 172.21.15.6:0/2303308613 >> 172.21.15.6:0/2303308613 conn(0x7ff41c0639d0 msgr2=0x7ff41c09dbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 -- 172.21.15.6:0/2303308613 shutdown_connections 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.197+0000 7ff4233eb700 1 -- 172.21.15.6:0/2303308613 wait complete. 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.198+0000 7f0a06105700 1 -- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 msgr2=0x7f0a00136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.198+0000 7f0a06105700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00136aa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0030c0 tx=0x7f09fc016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.198+0000 7ff4233eb700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.198+0000 7ff4233eb700 10 monclient: monmap: 2023-09-30T00:44:30.272 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.272 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:30.272 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:30.272 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.200+0000 7ff4233eb700 1 Processor -- start 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.200+0000 7ff4233eb700 1 -- start start 2023-09-30T00:44:30.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.200+0000 7ff4233eb700 10 monclient: init 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc01b450 con 0x7f0a00137b60 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: dump: 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.274 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09fc020070 con 0x7f0a00137b60 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 5 monclient: authenticate success, global_id 4263 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 10 monclient: _renew_subs 2023-09-30T00:44:30.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0009aee0 con 0x7f0a00137b60 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 10 monclient: _renew_subs 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a00136fe0 con 0x7f0a00137b60 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:30.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41c09baf0 0x7ff41c141870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): picked mon.noname-b con 0x7ff41c09baf0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09bec0 0x7ff41c06f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): picked mon.noname-c con 0x7ff41c09bec0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 0x7ff41c1318e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): picked mon.noname-a con 0x7ff41c133800 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c098650 con 0x7ff41c133800 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c097eb0 con 0x7ff41c09baf0 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff41c06c8a0 con 0x7ff41c09bec0 2023-09-30T00:44:30.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff4233eb700 10 monclient(hunting): authenticate will time out at 1266.498365s 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 0x7ff41c1318e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 1 -- 172.21.15.6:0/2911224515 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800e070 con 0x7f6e8009b350 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: dump: 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.278 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 1 -- 172.21.15.6:0/2911224515 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e780115a0 con 0x7f6e8009b350 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 1 -- 172.21.15.6:0/2911224515 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e7800bea0 con 0x7f6e8009b350 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e7effd700 10 monclient: dump: 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.280 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e867bf700 5 monclient: authenticate success, global_id 4268 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e867bf700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f6e867bf700 4 set_mon_vals no callback set 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f6e867bf700 10 monclient: shutdown 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f6e867bf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f6e867bf700 1 -- 172.21.15.6:0/2911224515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 msgr2=0x7f6e8009c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f6e867bf700 1 --2- 172.21.15.6:0/2911224515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e8009c9c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f6e7800b050 tx=0x7f6e78009780 comp rx=0 tx=0).stop 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 0x7ff41c1318e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42054/0 (socket says 172.21.15.6:42054) 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 1 -- 172.21.15.6:0/181262396 learned_addr learned my addr 172.21.15.6:0/181262396 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 10 monclient(hunting): get_auth_request con 0x7ff41c133800 auth_method 0 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7ff421988700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 10 monclient(hunting): handle_auth_done global_id 4266 payload 824 2023-09-30T00:44:30.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 1 -- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09bec0 msgr2=0x7ff41c06f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09bec0 0x7ff41c06f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 1 -- 172.21.15.6:0/181262396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41c09baf0 msgr2=0x7ff41c141870 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41c09baf0 0x7ff41c141870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 1 monclient: found mon.noname-a 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7ff421988700 1 -- 172.21.15.6:0/181262396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff40c0093c0 con 0x7ff41c133800 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09fc01b820 con 0x7f0a00137b60 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f5ffb700 10 monclient: dump: 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.284 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.201+0000 7f09f7fff700 4 set_mon_vals no callback set 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f0a07b68700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f0a07b68700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f0a07b68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.202+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e00050b0 con 0x7f0a00137b60 2023-09-30T00:44:30.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.203+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f09fc022030 con 0x7f0a00137b60 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f09f5ffb700 1 --2- 172.21.15.6:0/2224975489 >> v2:172.21.15.22:6812/142603 conn(0x7f09e40357f0 0x7f09e4037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f09fc049d10 con 0x7f0a00137b60 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f6e867bf700 1 -- 172.21.15.6:0/2911224515 shutdown_connections 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f6e867bf700 1 --2- 172.21.15.6:0/2911224515 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e8009c9c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f6e867bf700 1 -- 172.21.15.6:0/2911224515 >> 172.21.15.6:0/2911224515 conn(0x7f6e800a32b0 msgr2=0x7f6e800a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f0a05904700 1 --2- 172.21.15.6:0/2224975489 >> v2:172.21.15.22:6812/142603 conn(0x7f09e40357f0 0x7f09e4037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f0a05904700 10 monclient: get_auth_request con 0x7f09e40357f0 auth_method 0 2023-09-30T00:44:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7ff421988700 1 -- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 msgr2=0x7ff41c1318e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7ff421988700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 0x7ff41c1318e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff414009e70 tx=0x7ff41400a890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.204+0000 7f6e867bf700 1 -- 172.21.15.6:0/2911224515 shutdown_connections 2023-09-30T00:44:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.205+0000 7f6e867bf700 1 -- 172.21.15.6:0/2911224515 wait complete. 2023-09-30T00:44:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.205+0000 7f6e867bf700 10 monclient: build_initial_monmap 2023-09-30T00:44:30.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.205+0000 7f6e867bf700 10 monclient: monmap: 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.206+0000 7f6e867bf700 1 Processor -- start 2023-09-30T00:44:30.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f0a05904700 1 --2- 172.21.15.6:0/2224975489 >> v2:172.21.15.22:6812/142603 conn(0x7f09e40357f0 0x7f09e4037ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f09f8005c50 tx=0x7f09f8009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.206+0000 7f6e867bf700 1 -- start start 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.206+0000 7f6e867bf700 10 monclient: init 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e80143100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): picked mon.noname-c con 0x7f6e8009b350 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8009dc20 0x7f6e8011af60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): picked mon.noname-a con 0x7f6e8009dc20 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6e8009e740 0x7f6e8011b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): picked mon.noname-b con 0x7f6e8009e740 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e80069e00 con 0x7f6e8009dc20 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8011ec30 con 0x7f6e8009e740 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e8011e950 con 0x7f6e8009b350 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.207+0000 7f6e867bf700 10 monclient(hunting): authenticate will time out at 1266.504742s 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e80143100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e80143100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36616/0 (socket says 172.21.15.6:36616) 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 -- 172.21.15.6:0/1275201064 learned_addr learned my addr 172.21.15.6:0/1275201064 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): get_auth_request con 0x7f6e8009b350 auth_method 0 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8009dc20 0x7f6e8011af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): get_auth_request con 0x7f6e8009dc20 auth_method 0 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6e8009e740 0x7f6e8011b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): get_auth_request con 0x7f6e8009e740 auth_method 0 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:30.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e84d5c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient(hunting): handle_auth_done global_id 4274 payload 824 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 -- 172.21.15.6:0/1275201064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6e8009e740 msgr2=0x7f6e8011b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6e8009e740 0x7f6e8011b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 -- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8009dc20 msgr2=0x7f6e8011af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8009dc20 0x7f6e8011af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 monclient: found mon.noname-c 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.208+0000 7f6e7ffff700 1 -- 172.21.15.6:0/1275201064 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e78009a20 con 0x7f6e8009b350 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e7ffff700 1 -- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 msgr2=0x7f6e80143100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:30.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e7ffff700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e80143100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6e800685d0 tx=0x7f6e78019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78011d80 con 0x7f6e8009b350 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: dump: 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.295 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e78021070 con 0x7f6e8009b350 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e78026680 con 0x7f6e8009b350 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e77fff700 10 monclient: dump: 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.296 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e7e7fc700 4 set_mon_vals no callback set 2023-09-30T00:44:30.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f09fc010380 con 0x7f0a00137b60 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f09f5ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f09f5ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 5 monclient: authenticate success, global_id 4274 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 10 monclient: _renew_subs 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e8011e670 con 0x7f6e8009b350 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 10 monclient: _renew_subs 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.211+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e8011cb40 con 0x7f6e8009b350 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414007e50 con 0x7ff41c133800 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: dump: 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.299 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: _finish_auth 0 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:30.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff414010040 con 0x7ff41c133800 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff414014620 con 0x7ff41c133800 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4117fa700 10 monclient: dump: 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:30.302 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4137fe700 4 set_mon_vals no callback set 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 5 monclient: authenticate success, global_id 4266 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 10 monclient: _renew_subs 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c141db0 con 0x7ff41c133800 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 10 monclient: _renew_subs 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff41c06b3f0 con 0x7ff41c133800 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.214+0000 7ff4233eb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.214+0000 7ff4233eb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.214+0000 7ff4233eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.214+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff41c072b00 con 0x7ff41c133800 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.214+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff414014a50 con 0x7ff41c133800 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f6e867bf700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f6e867bf700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f6e867bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.213+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e80141f20 con 0x7f6e8009b350 2023-09-30T00:44:30.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.215+0000 7ff4117fa700 1 --2- 172.21.15.6:0/181262396 >> v2:172.21.15.22:6812/142603 conn(0x7ff4000357f0 0x7ff400037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.215+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff4140480a0 con 0x7ff41c133800 2023-09-30T00:44:30.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.215+0000 7ff421187700 1 --2- 172.21.15.6:0/181262396 >> v2:172.21.15.22:6812/142603 conn(0x7ff4000357f0 0x7ff400037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.215+0000 7ff421187700 10 monclient: get_auth_request con 0x7ff4000357f0 auth_method 0 2023-09-30T00:44:30.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.215+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6e78023030 con 0x7f6e8009b350 2023-09-30T00:44:30.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.216+0000 7f6e77fff700 1 --2- 172.21.15.6:0/1275201064 >> v2:172.21.15.22:6812/142603 conn(0x7f6e600357f0 0x7f6e60037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:30.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.216+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6e78049950 con 0x7f6e8009b350 2023-09-30T00:44:30.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.218+0000 7ff421187700 1 --2- 172.21.15.6:0/181262396 >> v2:172.21.15.22:6812/142603 conn(0x7ff4000357f0 0x7ff400037ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff40c006000 tx=0x7ff40c015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.220+0000 7f6e7f7fe700 1 --2- 172.21.15.6:0/1275201064 >> v2:172.21.15.22:6812/142603 conn(0x7f6e600357f0 0x7f6e60037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:30.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.220+0000 7f6e7f7fe700 10 monclient: get_auth_request con 0x7f6e600357f0 auth_method 0 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.222+0000 7f6e7f7fe700 1 --2- 172.21.15.6:0/1275201064 >> v2:172.21.15.22:6812/142603 conn(0x7f6e600357f0 0x7f6e60037ca0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6e70000c00 tx=0x7f6e70009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.224+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff4140795a0 con 0x7ff41c133800 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.225+0000 7ff4117fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.225+0000 7ff4117fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.226+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6e78012430 con 0x7f6e8009b350 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.226+0000 7f6e77fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:30.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.226+0000 7f6e77fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:30.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.390+0000 7f57eb022700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:30.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.390+0000 7f57eb022700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:30.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.390+0000 7f57eb022700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:30.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.390+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f57e4072b00 con 0x7f57e4132d00 2023-09-30T00:44:30.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.391+0000 7f57d8ff9700 1 -- 172.21.15.6:0/2281100517 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f57e00102e0 con 0x7f57e4132d00 2023-09-30T00:44:30.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.391+0000 7f57d8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:30.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.391+0000 7f57d8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:30.392 INFO:teuthology.orchestra.run.smithi006.stdout:47244640257 2023-09-30T00:44:30.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.393+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 >> v2:172.21.15.22:6812/142603 conn(0x7f57d00357d0 msgr2=0x7f57d0037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.393+0000 7f57eb022700 1 --2- 172.21.15.6:0/2281100517 >> v2:172.21.15.22:6812/142603 conn(0x7f57d00357d0 0x7f57d0037c80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009b80 tx=0x7f57d40091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.393+0000 7f57eb022700 10 monclient: shutdown 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.393+0000 7f57eb022700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.393+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 msgr2=0x7f57e4099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.393+0000 7f57eb022700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 0x7f57e4099290 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f57e000c4e0 tx=0x7f57e0007ea0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 shutdown_connections 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 --2- 172.21.15.6:0/2281100517 >> v2:172.21.15.22:6812/142603 conn(0x7f57d00357d0 0x7f57d0037c80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57e4138d40 0x7f57e40997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57e4098670 0x7f57e412f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 --2- 172.21.15.6:0/2281100517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57e4132d00 0x7f57e4099290 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 >> 172.21.15.6:0/2281100517 conn(0x7f57e40a5480 msgr2=0x7f57e41385d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 shutdown_connections 2023-09-30T00:44:30.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.394+0000 7f57eb022700 1 -- 172.21.15.6:0/2281100517 wait complete. 2023-09-30T00:44:30.407 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640257 for osd.5 2023-09-30T00:44:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.422+0000 7f0a07b68700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.422+0000 7f0a07b68700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.422+0000 7f0a07b68700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.422+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f09e0005630 con 0x7f0a00137b60 2023-09-30T00:44:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.424+0000 7f09f5ffb700 1 -- 172.21.15.6:0/2224975489 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f09fc010e60 con 0x7f0a00137b60 2023-09-30T00:44:30.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.424+0000 7f09f5ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:30.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.424+0000 7f09f5ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:30.424 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.426+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 >> v2:172.21.15.22:6812/142603 conn(0x7f09e40357f0 msgr2=0x7f09e4037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.426+0000 7f0a07b68700 1 --2- 172.21.15.6:0/2224975489 >> v2:172.21.15.22:6812/142603 conn(0x7f09e40357f0 0x7f09e4037ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f09f8005c50 tx=0x7f09f8009040 comp rx=0 tx=0).stop 2023-09-30T00:44:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.426+0000 7f0a07b68700 10 monclient: shutdown 2023-09-30T00:44:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.426+0000 7f0a07b68700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.426+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 msgr2=0x7f0a00136aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.426+0000 7f0a07b68700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00136aa0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f09fc0030c0 tx=0x7f09fc016000 comp rx=0 tx=0).stop 2023-09-30T00:44:30.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.429+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 shutdown_connections 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.429+0000 7f0a07b68700 1 --2- 172.21.15.6:0/2224975489 >> v2:172.21.15.22:6812/142603 conn(0x7f09e40357f0 0x7f09e4037ca0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.429+0000 7f0a07b68700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0a00137b60 0x7f0a00136aa0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.429+0000 7f0a07b68700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0a00098670 0x7f0a0009edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.429+0000 7f0a07b68700 1 --2- 172.21.15.6:0/2224975489 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0a00132d00 0x7f0a00141ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.430+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 >> 172.21.15.6:0/2224975489 conn(0x7f0a000a5480 msgr2=0x7f0a00131930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.430+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 shutdown_connections 2023-09-30T00:44:30.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.430+0000 7f0a07b68700 1 -- 172.21.15.6:0/2224975489 wait complete. 2023-09-30T00:44:30.442 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2023-09-30T00:44:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f786b6ec700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f786b6ec700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f786b6ec700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f7864072b00 con 0x7f7864098e90 2023-09-30T00:44:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f78597fa700 1 -- 172.21.15.6:0/3707059882 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f785c02b090 con 0x7f7864098e90 2023-09-30T00:44:30.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f78597fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:30.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.514+0000 7f78597fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:30.515 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:30.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.516+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 >> v2:172.21.15.22:6812/142603 conn(0x7f78480357a0 msgr2=0x7f7848037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.516+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3707059882 >> v2:172.21.15.22:6812/142603 conn(0x7f78480357a0 0x7f7848037c50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f78600093a0 tx=0x7f7860016000 comp rx=0 tx=0).stop 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.516+0000 7f786b6ec700 10 monclient: shutdown 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.516+0000 7f786b6ec700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.516+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 msgr2=0x7f786409d2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.516+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 0x7f786409d2e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f785c009e50 tx=0x7f785c00b040 comp rx=0 tx=0).stop 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 shutdown_connections 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3707059882 >> v2:172.21.15.22:6812/142603 conn(0x7f78480357a0 0x7f7848037c50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7864098e90 0x7f786409d2e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7864098670 0x7f786412c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 --2- 172.21.15.6:0/3707059882 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7864098a40 0x7f786409cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 >> 172.21.15.6:0/3707059882 conn(0x7f78640a5480 msgr2=0x7f7864130840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 shutdown_connections 2023-09-30T00:44:30.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.517+0000 7f786b6ec700 1 -- 172.21.15.6:0/3707059882 wait complete. 2023-09-30T00:44:30.530 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2023-09-30T00:44:30.542 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc91b652700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:30.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc91b652700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:30.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc91b652700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fc91411ae00 con 0x7fc91409a8f0 2023-09-30T00:44:30.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc9097fa700 1 -- 172.21.15.6:0/264974701 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc904033020 con 0x7fc91409a8f0 2023-09-30T00:44:30.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc9097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:30.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.543+0000 7fc9097fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:30.544 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:30.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 >> v2:172.21.15.22:6812/142603 conn(0x7fc8f80357f0 msgr2=0x7fc8f8037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 --2- 172.21.15.6:0/264974701 >> v2:172.21.15.22:6812/142603 conn(0x7fc8f80357f0 0x7fc8f8037ca0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fc914098fd0 tx=0x7fc910009060 comp rx=0 tx=0).stop 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 10 monclient: shutdown 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 msgr2=0x7fc91413b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 0x7fc91413b0c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc90400b3c0 tx=0x7fc904015000 comp rx=0 tx=0).stop 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 shutdown_connections 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 --2- 172.21.15.6:0/264974701 >> v2:172.21.15.22:6812/142603 conn(0x7fc8f80357f0 0x7fc8f8037ca0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc91413b6c0 0x7fc914097ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc91409a8f0 0x7fc91413b0c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 --2- 172.21.15.6:0/264974701 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc91409cd40 0x7fc9140977a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.545+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 >> 172.21.15.6:0/264974701 conn(0x7fc9140a32b0 msgr2=0x7fc91409db20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.546+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 shutdown_connections 2023-09-30T00:44:30.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.546+0000 7fc91b652700 1 -- 172.21.15.6:0/264974701 wait complete. 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4233eb700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4233eb700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4233eb700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff41c067f70 con 0x7ff41c133800 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4117fa700 1 -- 172.21.15.6:0/181262396 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff414014d00 con 0x7ff41c133800 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4117fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:30.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.558+0000 7ff4117fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:30.559 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:30.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e867bf700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:30.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e867bf700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:30.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e867bf700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:30.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6e80072b00 con 0x7f6e8009b350 2023-09-30T00:44:30.561 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2023-09-30T00:44:30.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e77fff700 1 -- 172.21.15.6:0/1275201064 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6e7801a3b0 con 0x7f6e8009b350 2023-09-30T00:44:30.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e77fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:30.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.559+0000 7f6e77fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:30.562 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:30.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 >> v2:172.21.15.22:6812/142603 conn(0x7ff4000357f0 msgr2=0x7ff400037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 --2- 172.21.15.6:0/181262396 >> v2:172.21.15.22:6812/142603 conn(0x7ff4000357f0 0x7ff400037ca0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff40c006000 tx=0x7ff40c015000 comp rx=0 tx=0).stop 2023-09-30T00:44:30.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 10 monclient: shutdown 2023-09-30T00:44:30.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 msgr2=0x7ff41c1318e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 0x7ff41c1318e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ff414009e70 tx=0x7ff41400a890 comp rx=0 tx=0).stop 2023-09-30T00:44:30.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 shutdown_connections 2023-09-30T00:44:30.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 --2- 172.21.15.6:0/181262396 >> v2:172.21.15.22:6812/142603 conn(0x7ff4000357f0 0x7ff400037ca0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff41c133800 0x7ff41c1318e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff41c09baf0 0x7ff41c141870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 --2- 172.21.15.6:0/181262396 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff41c09bec0 0x7ff41c06f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 >> 172.21.15.6:0/181262396 conn(0x7ff41c0639d0 msgr2=0x7ff41c09a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 shutdown_connections 2023-09-30T00:44:30.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7ff4233eb700 1 -- 172.21.15.6:0/181262396 wait complete. 2023-09-30T00:44:30.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 >> v2:172.21.15.22:6812/142603 conn(0x7f6e600357f0 msgr2=0x7f6e60037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.561+0000 7f6e867bf700 1 --2- 172.21.15.6:0/1275201064 >> v2:172.21.15.22:6812/142603 conn(0x7f6e600357f0 0x7f6e60037ca0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6e70000c00 tx=0x7f6e70009040 comp rx=0 tx=0).stop 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 10 monclient: shutdown 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 msgr2=0x7f6e80143100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e80143100 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f6e800685d0 tx=0x7f6e78019000 comp rx=0 tx=0).stop 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 shutdown_connections 2023-09-30T00:44:30.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 --2- 172.21.15.6:0/1275201064 >> v2:172.21.15.22:6812/142603 conn(0x7f6e600357f0 0x7f6e60037ca0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6e8009e740 0x7f6e8011b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6e8009b350 0x7f6e80143100 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 --2- 172.21.15.6:0/1275201064 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6e8009dc20 0x7f6e8011af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 >> 172.21.15.6:0/1275201064 conn(0x7f6e800a32b0 msgr2=0x7f6e8006a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 shutdown_connections 2023-09-30T00:44:30.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:30.562+0000 7f6e867bf700 1 -- 172.21.15.6:0/1275201064 wait complete. 2023-09-30T00:44:30.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2023-09-30T00:44:30.576 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2023-09-30T00:44:31.408 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:44:31.444 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:44:31.531 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.555+0000 7f0fc9947700 10 monclient: get_monmap_and_config 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.555+0000 7f0fc9947700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.555+0000 7f0fc9947700 10 monclient: monmap: 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:31.555549+0000 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:31.555549+0000 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.555 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:31.556 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:31.556 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.555+0000 7f0fc9947700 1 Processor -- start 2023-09-30T00:44:31.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 1 -- start start 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient: init 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f0fc4098670 0x7f0fc4132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): picked mon.noname-c con 0x7f0fc4098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0fc4098a40 0x7f0fc412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): picked mon.noname-a con 0x7f0fc4098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f0fc4098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0fc40a1000 con 0x7f0fc4098e90 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc40a0a50 con 0x7f0fc4098a40 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0fc40a0730 con 0x7f0fc4098670 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc9947700 10 monclient(hunting): authenticate will time out at 1267.853676s 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0fc4098a40 0x7f0fc412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc37fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f0fc4098e90 0x7f0fc412f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60900/0 (socket says 172.21.15.6:60900) 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc37fe700 1 -- 172.21.15.6:0/4058394838 learned_addr learned my addr 172.21.15.6:0/4058394838 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): get_auth_request con 0x7f0fc4098a40 auth_method 0 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.556+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc2ffd700 1 --2- 172.21.15.6:0/4058394838 >> v2:172.21.15.22:3300/0 conn(0x7f0fc4098670 0x7f0fc4132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc2ffd700 10 monclient(hunting): get_auth_request con 0x7f0fc4098670 auth_method 0 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_done global_id 4272 payload 274 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 1 -- 172.21.15.6:0/4058394838 >> v2:172.21.15.22:3300/0 conn(0x7f0fc4098670 msgr2=0x7f0fc4132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 1 --2- 172.21.15.6:0/4058394838 >> v2:172.21.15.22:3300/0 conn(0x7f0fc4098670 0x7f0fc4132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 1 -- 172.21.15.6:0/4058394838 >> v1:172.21.15.6:6789/0 conn(0x7f0fc4098e90 legacy=0x7f0fc412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 1 monclient: found mon.noname-a 2023-09-30T00:44:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc27fc700 1 -- 172.21.15.6:0/4058394838 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fc4068df0 con 0x7f0fc4098a40 2023-09-30T00:44:31.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.557+0000 7f0fc2ffd700 1 --2- 172.21.15.6:0/4058394838 >> v2:172.21.15.22:3300/0 conn(0x7f0fc4098670 0x7f0fc4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:31.562 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:44:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc27fc700 1 -- 172.21.15.6:0/4058394838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098a40 msgr2=0x7f0fc412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc27fc700 1 --2- 172.21.15.6:0/4058394838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098a40 0x7f0fc412ede0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0fc409fe60 tx=0x7f0fb80091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23ee5a0d5afc5d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 1 -- 172.21.15.6:0/4058394838 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb800b880 con 0x7f0fc4098a40 2023-09-30T00:44:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: dump: 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 1 -- 172.21.15.6:0/4058394838 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb800b9e0 con 0x7f0fc4098a40 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc9947700 5 monclient: authenticate success, global_id 4272 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc9947700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc9947700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc9947700 4 set_mon_vals no callback set 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 1 -- 172.21.15.6:0/4058394838 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb800bd80 con 0x7f0fc4098a40 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:31.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.559+0000 7f0fc1ffb700 10 monclient: dump: 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 10 monclient: shutdown 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 -- 172.21.15.6:0/4058394838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098a40 msgr2=0x7f0fc412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 --2- 172.21.15.6:0/4058394838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098a40 0x7f0fc412ede0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f0fc409fe60 tx=0x7f0fb80091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 -- 172.21.15.6:0/4058394838 shutdown_connections 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 --2- 172.21.15.6:0/4058394838 >> v2:172.21.15.22:3300/0 conn(0x7f0fc4098670 0x7f0fc4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 --2- 172.21.15.6:0/4058394838 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098a40 0x7f0fc412ede0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 -- 172.21.15.6:0/4058394838 >> 172.21.15.6:0/4058394838 conn(0x7f0fc40a5480 msgr2=0x7f0fc40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 -- 172.21.15.6:0/4058394838 shutdown_connections 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.560+0000 7f0fc9947700 1 -- 172.21.15.6:0/4058394838 wait complete. 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.561+0000 7f0fc9947700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.561+0000 7f0fc9947700 10 monclient: monmap: 2023-09-30T00:44:31.572 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 Processor -- start 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 -- start start 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient: init 2023-09-30T00:44:31.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0fc4098670 0x7f0fc409f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): picked mon.noname-b con 0x7f0fc4098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 0x7f0fc41419e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): picked mon.noname-c con 0x7f0fc4098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098e90 0x7f0fc4141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): picked mon.noname-a con 0x7f0fc4098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc40a1570 con 0x7f0fc4098e90 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc409e670 con 0x7f0fc4098670 2023-09-30T00:44:31.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0fc409d1c0 con 0x7f0fc4098a40 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.562+0000 7f0fc9947700 10 monclient(hunting): authenticate will time out at 1267.859991s 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 0x7f0fc41419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 0x7f0fc41419e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36624/0 (socket says 172.21.15.6:36624) 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 -- 172.21.15.6:0/944242941 learned_addr learned my addr 172.21.15.6:0/944242941 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): get_auth_request con 0x7f0fc4098a40 auth_method 0 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098e90 0x7f0fc4141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0fc4098670 0x7f0fc409f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): get_auth_request con 0x7f0fc4098670 auth_method 0 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): get_auth_request con 0x7f0fc4098e90 auth_method 0 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient(hunting): handle_auth_done global_id 4277 payload 824 2023-09-30T00:44:31.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 -- 172.21.15.6:0/944242941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0fc4098670 msgr2=0x7f0fc409f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0fc4098670 0x7f0fc409f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 -- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098e90 msgr2=0x7f0fc4141f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098e90 0x7f0fc4141f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc27fc700 1 -- 172.21.15.6:0/944242941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fb8009420 con 0x7f0fc4098a40 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.563+0000 7f0fc37fe700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098e90 0x7f0fc4141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:31.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.565+0000 7f0fc27fc700 1 -- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 msgr2=0x7f0fc41419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.565+0000 7f0fc27fc700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 0x7f0fc41419e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0fc406a6b0 tx=0x7f0fb8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb8000d10 con 0x7f0fc4098a40 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: dump: 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.579 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0fb8020070 con 0x7f0fc4098a40 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0fb801b450 con 0x7f0fc4098a40 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 5 monclient: authenticate success, global_id 4277 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 10 monclient: _renew_subs 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fc409e0b0 con 0x7f0fc4098a40 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0faaffd700 10 monclient: dump: 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.581 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc17fa700 4 set_mon_vals no callback set 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 10 monclient: _renew_subs 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.566+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fc409f570 con 0x7f0fc4098a40 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0fb8022030 con 0x7f0fc4098a40 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0fc9947700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0fc9947700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0fc9947700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fc40982d0 con 0x7f0fc4098a40 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0faaffd700 1 --2- 172.21.15.6:0/944242941 >> v2:172.21.15.22:6812/142603 conn(0x7f0fb00357a0 0x7f0fb0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.567+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0fb8049640 con 0x7f0fc4098a40 2023-09-30T00:44:31.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.568+0000 7f0fc2ffd700 1 --2- 172.21.15.6:0/944242941 >> v2:172.21.15.22:6812/142603 conn(0x7f0fb00357a0 0x7f0fb0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.568+0000 7f0fc2ffd700 10 monclient: get_auth_request con 0x7f0fb00357a0 auth_method 0 2023-09-30T00:44:31.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.570+0000 7f0fc2ffd700 1 --2- 172.21.15.6:0/944242941 >> v2:172.21.15.22:6812/142603 conn(0x7f0fb00357a0 0x7f0fb0037c50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4006f00 tx=0x7f0fb4008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.573+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0fb801ba20 con 0x7f0fc4098a40 2023-09-30T00:44:31.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.573+0000 7f0faaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.573+0000 7f0faaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:31.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:44:31.585 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:44:31.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.600+0000 7f3015e7e700 10 monclient: get_monmap_and_config 2023-09-30T00:44:31.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.600+0000 7f3015e7e700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.600+0000 7f3015e7e700 10 monclient: monmap: 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:31.600823+0000 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:31.600823+0000 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:31.601 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.601+0000 7f3015e7e700 1 Processor -- start 2023-09-30T00:44:31.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.601+0000 7f3015e7e700 1 -- start start 2023-09-30T00:44:31.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.601+0000 7f3015e7e700 10 monclient: init 2023-09-30T00:44:31.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-09-30T00:44:31.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3010098670 0x7f3010132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): picked mon.noname-c con 0x7f3010098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3010098a40 0x7f301012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): picked mon.noname-a con 0x7f3010098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3010098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30100a1000 con 0x7f3010098e90 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30100a0a50 con 0x7f3010098a40 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30100a0730 con 0x7f3010098670 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f3015e7e700 10 monclient(hunting): authenticate will time out at 1267.899388s 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300ffff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3010098e90 0x7f301012f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60912/0 (socket says 172.21.15.6:60912) 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300ffff700 1 -- 172.21.15.6:0/48315549 learned_addr learned my addr 172.21.15.6:0/48315549 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 1 --2- 172.21.15.6:0/48315549 >> v2:172.21.15.22:3300/0 conn(0x7f3010098670 0x7f3010132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): get_auth_request con 0x7f3010098670 auth_method 0 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 1 --2- 172.21.15.6:0/48315549 >> v2:172.21.15.6:3300/0 conn(0x7f3010098a40 0x7f301012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): get_auth_request con 0x7f3010098a40 auth_method 0 2023-09-30T00:44:31.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.607 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.607 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.607 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.607 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.607 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.602+0000 7f300effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.609 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 10 monclient(hunting): handle_auth_done global_id 4282 payload 274 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 1 -- 172.21.15.6:0/48315549 >> v2:172.21.15.6:3300/0 conn(0x7f3010098a40 msgr2=0x7f301012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 1 --2- 172.21.15.6:0/48315549 >> v2:172.21.15.6:3300/0 conn(0x7f3010098a40 0x7f301012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 1 -- 172.21.15.6:0/48315549 >> v1:172.21.15.6:6789/0 conn(0x7f3010098e90 legacy=0x7f301012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.603+0000 7f300f7fe700 1 -- 172.21.15.6:0/48315549 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010068df0 con 0x7f3010098670 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.606+0000 7f300f7fe700 1 -- 172.21.15.6:0/48315549 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098670 msgr2=0x7f3010132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.610 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.606+0000 7f300f7fe700 1 --2- 172.21.15.6:0/48315549 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098670 0x7f3010132d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f301009fe60 tx=0x7f30000091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dbbe0c7dc570a9e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 1 -- 172.21.15.6:0/48315549 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000011840 con 0x7f3010098670 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: dump: 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.611 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 1 -- 172.21.15.6:0/48315549 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30000119a0 con 0x7f3010098670 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f3015e7e700 5 monclient: authenticate success, global_id 4282 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f3015e7e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f3015e7e700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 1 -- 172.21.15.6:0/48315549 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000011d40 con 0x7f3010098670 2023-09-30T00:44:31.612 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f3015e7e700 4 set_mon_vals no callback set 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.609+0000 7f300e7fc700 10 monclient: dump: 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.613 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.610+0000 7f3015e7e700 10 monclient: shutdown 2023-09-30T00:44:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.610+0000 7f3015e7e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.610+0000 7f3015e7e700 1 -- 172.21.15.6:0/48315549 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098670 msgr2=0x7f3010132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.614 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.610+0000 7f3015e7e700 1 --2- 172.21.15.6:0/48315549 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098670 0x7f3010132d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f301009fe60 tx=0x7f30000091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.611+0000 7f3015e7e700 1 -- 172.21.15.6:0/48315549 shutdown_connections 2023-09-30T00:44:31.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.611+0000 7f3015e7e700 1 --2- 172.21.15.6:0/48315549 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098670 0x7f3010132d00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.611+0000 7f3015e7e700 1 --2- 172.21.15.6:0/48315549 >> v2:172.21.15.6:3300/0 conn(0x7f3010098a40 0x7f301012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.611+0000 7f3015e7e700 1 -- 172.21.15.6:0/48315549 >> 172.21.15.6:0/48315549 conn(0x7f30100a5480 msgr2=0x7f30100a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.611+0000 7f3015e7e700 1 -- 172.21.15.6:0/48315549 shutdown_connections 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.611+0000 7f3015e7e700 1 -- 172.21.15.6:0/48315549 wait complete. 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.612+0000 7f3015e7e700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.612+0000 7f3015e7e700 10 monclient: monmap: 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.616 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.612+0000 7f3015e7e700 1 Processor -- start 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 -- start start 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient: init 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3010098670 0x7f301009d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): picked mon.noname-a con 0x7f3010098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 0x7f301009d570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): picked mon.noname-c con 0x7f3010098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098e90 0x7f301009dab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): picked mon.noname-b con 0x7f3010098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301006bdd0 con 0x7f3010098670 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3010131330 con 0x7f3010098e90 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f301006b7d0 con 0x7f3010098a40 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.613+0000 7f3015e7e700 10 monclient(hunting): authenticate will time out at 1267.910939s 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098e90 0x7f301009dab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098e90 0x7f301009dab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.6:49404/0 (socket says 172.21.15.6:49404) 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 1 -- 172.21.15.6:0/458637215 learned_addr learned my addr 172.21.15.6:0/458637215 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): get_auth_request con 0x7f3010098e90 auth_method 0 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 0x7f301009d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): get_auth_request con 0x7f3010098a40 auth_method 0 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3010098670 0x7f301009d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): get_auth_request con 0x7f3010098670 auth_method 0 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.614+0000 7f300f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 10 monclient(hunting): handle_auth_done global_id 4280 payload 824 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 1 -- 172.21.15.6:0/458637215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098e90 msgr2=0x7f301009dab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098e90 0x7f301009dab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 1 -- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3010098670 msgr2=0x7f301009d030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3010098670 0x7f301009d030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.615+0000 7f300effd700 1 -- 172.21.15.6:0/458637215 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3000009420 con 0x7f3010098a40 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f300effd700 1 -- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 msgr2=0x7f301009d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f300effd700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 0x7f301009d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f300400ea50 tx=0x7f30040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300400ee90 con 0x7f3010098a40 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f2ff77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f2ff77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f2ff77fe700 10 monclient: dump: 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.624 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f2ff77fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f2ff77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f3015e7e700 5 monclient: authenticate success, global_id 4280 2023-09-30T00:44:31.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f3015e7e700 10 monclient: _renew_subs 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f3015e7e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.617+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301006b1b0 con 0x7f3010098a40 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f3015e7e700 10 monclient: _renew_subs 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f3015e7e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301009f610 con 0x7f3010098a40 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f300400ca00 con 0x7f3010098a40 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f2ff77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300400cdc0 con 0x7f3010098a40 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f2ff77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f2ff77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:31.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f2ff77fe700 10 monclient: dump: 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.618+0000 7f300dffb700 4 set_mon_vals no callback set 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.619+0000 7f3015e7e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.619+0000 7f3015e7e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.619+0000 7f3015e7e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.619+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ff00050b0 con 0x7f3010098a40 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.620+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3004013680 con 0x7f3010098a40 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.620+0000 7f2ff77fe700 1 --2- 172.21.15.6:0/458637215 >> v2:172.21.15.22:6812/142603 conn(0x7f2ffc0357a0 0x7f2ffc037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.620+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3004048240 con 0x7f3010098a40 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.620+0000 7f300f7fe700 1 --2- 172.21.15.6:0/458637215 >> v2:172.21.15.22:6812/142603 conn(0x7f2ffc0357a0 0x7f2ffc037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.620+0000 7f300f7fe700 10 monclient: get_auth_request con 0x7f2ffc0357a0 auth_method 0 2023-09-30T00:44:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.624+0000 7f300f7fe700 1 --2- 172.21.15.6:0/458637215 >> v2:172.21.15.22:6812/142603 conn(0x7f2ffc0357a0 0x7f2ffc037c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30000093a0 tx=0x7f3000016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.629+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3004024760 con 0x7f3010098a40 2023-09-30T00:44:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.629+0000 7f2ff77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.629+0000 7f2ff77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:31.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.722+0000 7f3715931700 10 monclient: get_monmap_and_config 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.722+0000 7f3715931700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.722+0000 7f3715931700 10 monclient: monmap: 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:31.723105+0000 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:31.723105+0000 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:31.723 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:31.724 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:31.724 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:31.724 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.723+0000 7f3715931700 1 Processor -- start 2023-09-30T00:44:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.723+0000 7f3715931700 1 -- start start 2023-09-30T00:44:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.723+0000 7f3715931700 10 monclient: init 2023-09-30T00:44:31.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3710098670 0x7f3710132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient(hunting): picked mon.noname-c con 0x7f3710098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3710098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3710098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37100a1000 con 0x7f3710098e90 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f371012fc00 con 0x7f3710098a40 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37100a0a50 con 0x7f3710098670 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.724+0000 7f3715931700 10 monclient(hunting): authenticate will time out at 1268.021859s 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370e7fc700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7f3710098a40 0x7f371012ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57364/0 (socket says 172.21.15.6:57364) 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3710098670 0x7f3710132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370e7fc700 1 -- 172.21.15.6:0/480722184 learned_addr learned my addr 172.21.15.6:0/480722184 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): get_auth_request con 0x7f3710098670 auth_method 0 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient(hunting): handle_auth_done global_id 4288 payload 274 2023-09-30T00:44:31.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 1 -- 172.21.15.6:0/480722184 >> v1:172.21.15.22:6789/0 conn(0x7f3710098a40 legacy=0x7f371012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 1 -- 172.21.15.6:0/480722184 >> v1:172.21.15.6:6789/0 conn(0x7f3710098e90 legacy=0x7f371012f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:31.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.725+0000 7f370effd700 1 -- 172.21.15.6:0/480722184 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37100a0730 con 0x7f3710098670 2023-09-30T00:44:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370effd700 1 -- 172.21.15.6:0/480722184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 msgr2=0x7f3710132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370effd700 1 --2- 172.21.15.6:0/480722184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 0x7f3710132d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37040098e0 tx=0x7f37040091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dfbfd251d61637b4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 1 -- 172.21.15.6:0/480722184 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370400b9f0 con 0x7f3710098670 2023-09-30T00:44:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: dump: 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 1 -- 172.21.15.6:0/480722184 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370400bb50 con 0x7f3710098670 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 1 -- 172.21.15.6:0/480722184 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3704011420 con 0x7f3710098670 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f370dffb700 10 monclient: dump: 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.731 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f3715931700 5 monclient: authenticate success, global_id 4288 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f3715931700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.728+0000 7f3715931700 4 set_mon_vals no callback set 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 10 monclient: shutdown 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 -- 172.21.15.6:0/480722184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 msgr2=0x7f3710132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 --2- 172.21.15.6:0/480722184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 0x7f3710132d00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f37040098e0 tx=0x7f37040091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 -- 172.21.15.6:0/480722184 shutdown_connections 2023-09-30T00:44:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 --2- 172.21.15.6:0/480722184 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 0x7f3710132d00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 -- 172.21.15.6:0/480722184 >> 172.21.15.6:0/480722184 conn(0x7f37100a5480 msgr2=0x7f37100a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 -- 172.21.15.6:0/480722184 shutdown_connections 2023-09-30T00:44:31.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.729+0000 7f3715931700 1 -- 172.21.15.6:0/480722184 wait complete. 2023-09-30T00:44:31.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.730+0000 7f3715931700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.730+0000 7f3715931700 10 monclient: monmap: 2023-09-30T00:44:31.734 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.731+0000 7f3715931700 1 Processor -- start 2023-09-30T00:44:31.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.731+0000 7f3715931700 1 -- start start 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.731+0000 7f3715931700 10 monclient: init 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 0x7f3710099940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): picked mon.noname-b con 0x7f3710098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3710098a40 0x7f3710099e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): picked mon.noname-c con 0x7f3710098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 0x7f371009a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): picked mon.noname-a con 0x7f3710098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f371006e900 con 0x7f3710098e90 2023-09-30T00:44:31.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37100a1570 con 0x7f3710098670 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f371006b670 con 0x7f3710098a40 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f3715931700 10 monclient(hunting): authenticate will time out at 1268.029449s 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 0x7f371009a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 0x7f371009a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42100/0 (socket says 172.21.15.6:42100) 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 1 -- 172.21.15.6:0/2133406804 learned_addr learned my addr 172.21.15.6:0/2133406804 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 10 monclient(hunting): get_auth_request con 0x7f3710098e90 auth_method 0 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.732+0000 7f370f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370e7fc700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3710098a40 0x7f3710099e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370e7fc700 10 monclient(hunting): get_auth_request con 0x7f3710098a40 auth_method 0 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 10 monclient(hunting): handle_auth_done global_id 4284 payload 824 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 1 -- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3710098a40 msgr2=0x7f3710099e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3710098a40 0x7f3710099e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 1 -- 172.21.15.6:0/2133406804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 msgr2=0x7f3710099940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 0x7f3710099940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 1 monclient: found mon.noname-a 2023-09-30T00:44:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.733+0000 7f370f7fe700 1 -- 172.21.15.6:0/2133406804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3704009420 con 0x7f3710098e90 2023-09-30T00:44:31.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.735+0000 7f370f7fe700 1 -- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 msgr2=0x7f371009a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.735+0000 7f370f7fe700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 0x7f371009a3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f370800c930 tx=0x7f37080026e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370800ce70 con 0x7f3710098e90 2023-09-30T00:44:31.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: dump: 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.827 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f370800a9f0 con 0x7f3710098e90 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f370800add0 con 0x7f3710098e90 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f36f6ffd700 10 monclient: dump: 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.828 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 5 monclient: authenticate success, global_id 4284 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 10 monclient: _renew_subs 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f371006b390 con 0x7f3710098e90 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 10 monclient: _renew_subs 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.736+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f371009b310 con 0x7f3710098e90 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.738+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3708012630 con 0x7f3710098e90 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.738+0000 7f36f6ffd700 1 --2- 172.21.15.6:0/2133406804 >> v2:172.21.15.22:6812/142603 conn(0x7f36fc0357f0 0x7f36fc037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.738+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3708047780 con 0x7f3710098e90 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.738+0000 7f370effd700 1 --2- 172.21.15.6:0/2133406804 >> v2:172.21.15.22:6812/142603 conn(0x7f36fc0357f0 0x7f36fc037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.738+0000 7f370effd700 10 monclient: get_auth_request con 0x7f36fc0357f0 auth_method 0 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.739+0000 7f370d7fa700 4 set_mon_vals no callback set 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.740+0000 7f3715931700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f3715931700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f3715931700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36f00050b0 con 0x7f3710098e90 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f5220429700 10 monclient: get_monmap_and_config 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f5220429700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f5220429700 10 monclient: monmap: 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:31.741547+0000 2023-09-30T00:44:31.831 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:31.741547+0000 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.741+0000 7f370effd700 1 --2- 172.21.15.6:0/2133406804 >> v2:172.21.15.22:6812/142603 conn(0x7f36fc0357f0 0x7f36fc037ca0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f37040093a0 tx=0x7f3704016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.742+0000 7f5220429700 1 Processor -- start 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.742+0000 7f5220429700 1 -- start start 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.742+0000 7f5220429700 10 monclient: init 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5218098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:31.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5218132d00 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5218138d40 0x7f521813b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient(hunting): picked mon.noname-c con 0x7f5218138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180a1000 con 0x7f5218098670 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52180a0730 con 0x7f5218132d00 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5218099150 con 0x7f5218138d40 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f5220429700 10 monclient(hunting): authenticate will time out at 1268.040491s 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f521e1c5700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5218098670 0x7f5218098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60920/0 (socket says 172.21.15.6:60920) 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.743+0000 7f521e1c5700 1 -- 172.21.15.6:0/1318122391 learned_addr learned my addr 172.21.15.6:0/1318122391 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:31.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 1 --2- 172.21.15.6:0/1318122391 >> v2:172.21.15.22:3300/0 conn(0x7f5218138d40 0x7f521813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): get_auth_request con 0x7f5218138d40 auth_method 0 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.744+0000 7f521e9c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 10 monclient(hunting): handle_auth_done global_id 4291 payload 274 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 1 -- 172.21.15.6:0/1318122391 >> v1:172.21.15.22:6789/0 conn(0x7f5218132d00 legacy=0x7f5218137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 1 -- 172.21.15.6:0/1318122391 >> v1:172.21.15.6:6789/0 conn(0x7f5218098670 legacy=0x7f5218098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.745+0000 7f521e9c6700 1 -- 172.21.15.6:0/1318122391 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5218068df0 con 0x7f5218138d40 2023-09-30T00:44:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.746+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3708016080 con 0x7f3710098e90 2023-09-30T00:44:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.746+0000 7f36f6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.746+0000 7f36f6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:31.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.748+0000 7f521e9c6700 1 -- 172.21.15.6:0/1318122391 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 msgr2=0x7f521813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.748+0000 7f521e9c6700 1 --2- 172.21.15.6:0/1318122391 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521813b120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5210009e60 tx=0x7f5210009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7536b9f0e3231c9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 1 -- 172.21.15.6:0/1318122391 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521000e070 con 0x7f5218138d40 2023-09-30T00:44:31.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: dump: 2023-09-30T00:44:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.841 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 1 -- 172.21.15.6:0/1318122391 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5210012430 con 0x7f5218138d40 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 1 -- 172.21.15.6:0/1318122391 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f52100127d0 con 0x7f5218138d40 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f521d1c3700 10 monclient: dump: 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.843 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f5220429700 5 monclient: authenticate success, global_id 4291 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f5220429700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.750+0000 7f5220429700 4 set_mon_vals no callback set 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 10 monclient: shutdown 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 -- 172.21.15.6:0/1318122391 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 msgr2=0x7f521813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 --2- 172.21.15.6:0/1318122391 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521813b120 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f5210009e60 tx=0x7f5210009780 comp rx=0 tx=0).stop 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 -- 172.21.15.6:0/1318122391 shutdown_connections 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 --2- 172.21.15.6:0/1318122391 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521813b120 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.844 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 -- 172.21.15.6:0/1318122391 >> 172.21.15.6:0/1318122391 conn(0x7f52180a5480 msgr2=0x7f52180a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 -- 172.21.15.6:0/1318122391 shutdown_connections 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.751+0000 7f5220429700 1 -- 172.21.15.6:0/1318122391 wait complete. 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.752+0000 7f5220429700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.752+0000 7f5220429700 10 monclient: monmap: 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:31.845 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.752+0000 7f5220429700 1 Processor -- start 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.753+0000 7f5220429700 1 -- start start 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.753+0000 7f5220429700 10 monclient: init 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5218098670 0x7f521806e5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): picked mon.noname-a con 0x7f5218098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 0x7f521812f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): picked mon.noname-c con 0x7f5218132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521806d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): picked mon.noname-b con 0x7f5218138d40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521806ae90 con 0x7f5218098670 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f521806b3c0 con 0x7f5218138d40 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52180a1500 con 0x7f5218132d00 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f5220429700 10 monclient(hunting): authenticate will time out at 1268.051256s 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5218098670 0x7f521806e5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5218098670 0x7f521806e5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42110/0 (socket says 172.21.15.6:42110) 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 1 -- 172.21.15.6:0/929499478 learned_addr learned my addr 172.21.15.6:0/929499478 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 10 monclient(hunting): get_auth_request con 0x7f5218098670 auth_method 0 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e1c5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.754+0000 7f521e9c6700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521806d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): get_auth_request con 0x7f5218138d40 auth_method 0 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 0x7f521812f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): get_auth_request con 0x7f5218132d00 auth_method 0 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e9c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e1c5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e1c5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521e1c5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient(hunting): handle_auth_done global_id 4286 payload 824 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 -- 172.21.15.6:0/929499478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 msgr2=0x7f521806d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521806d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 -- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5218098670 msgr2=0x7f521806e5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5218098670 0x7f521806e5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.755+0000 7f521d9c4700 1 -- 172.21.15.6:0/929499478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210009a20 con 0x7f5218132d00 2023-09-30T00:44:31.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.757+0000 7f521d9c4700 1 -- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 msgr2=0x7f521812f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.757+0000 7f521d9c4700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 0x7f521812f890 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f521400c510 tx=0x7f5214007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f521400ce80 con 0x7f5218132d00 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: dump: 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.851 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52140117a0 con 0x7f5218132d00 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 5 monclient: authenticate success, global_id 4286 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 10 monclient: _renew_subs 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5218130420 con 0x7f5218132d00 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5214011b60 con 0x7f5218132d00 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f520e7fc700 10 monclient: dump: 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.853 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f521c9c2700 4 set_mon_vals no callback set 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 10 monclient: _renew_subs 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.758+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f521812f320 con 0x7f5218132d00 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.759+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f521401c460 con 0x7f5218132d00 2023-09-30T00:44:31.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.760+0000 7f520e7fc700 1 --2- 172.21.15.6:0/929499478 >> v2:172.21.15.22:6812/142603 conn(0x7f51fc0357f0 0x7f51fc037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.760+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5214047540 con 0x7f5218132d00 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.760+0000 7f5220429700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.760+0000 7f5220429700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.760+0000 7f5220429700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.760+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f521806b220 con 0x7f5218132d00 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.762+0000 7f521e1c5700 1 --2- 172.21.15.6:0/929499478 >> v2:172.21.15.22:6812/142603 conn(0x7f51fc0357f0 0x7f51fc037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.762+0000 7f521e1c5700 10 monclient: get_auth_request con 0x7f51fc0357f0 auth_method 0 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.764+0000 7f521e1c5700 1 --2- 172.21.15.6:0/929499478 >> v2:172.21.15.22:6812/142603 conn(0x7f51fc0357f0 0x7f51fc037ca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5208007880 tx=0x7f5208004a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.766+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f521806b220 con 0x7f5218132d00 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.766+0000 7f520e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.766+0000 7f520e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.776+0000 7f4ee0233700 10 monclient: get_monmap_and_config 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.776+0000 7f4ee0233700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.776+0000 7f4ee0233700 10 monclient: monmap: 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:31.776756+0000 2023-09-30T00:44:31.859 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:31.776756+0000 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.777+0000 7f4ee0233700 1 Processor -- start 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.777+0000 7f4ee0233700 1 -- start start 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.777+0000 7f4ee0233700 10 monclient: init 2023-09-30T00:44:31.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient: _add_conns ranks=[4,3,2,1,0] 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4ed80986e0 0x7f4ed8098ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): picked mon.noname-c con 0x7f4ed80986e0 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4ed8136f30 0x7f4ed8068df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): picked mon.noname-a con 0x7f4ed8136f30 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8137330 0x7f4ed8069330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): picked mon.noname-b con 0x7f4ed8137330 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed806a260 con 0x7f4ed8136f30 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ed8096e60 con 0x7f4ed80986e0 2023-09-30T00:44:31.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80a09e0 con 0x7f4ed8137330 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4ee0233700 10 monclient(hunting): authenticate will time out at 1268.075744s 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.778+0000 7f4eddfcf700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4ed80986e0 0x7f4ed8098ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4ed80986e0 0x7f4ed8098ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.22:3300/0 says I am v2:172.21.15.6:49444/0 (socket says 172.21.15.6:49444) 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3741273817 learned_addr learned my addr 172.21.15.6:0/3741273817 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): get_auth_request con 0x7f4ed80986e0 auth_method 0 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4ede7d0700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8137330 0x7f4ed8069330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4ede7d0700 10 monclient(hunting): get_auth_request con 0x7f4ed8137330 auth_method 0 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4ede7d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4ede7d0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4ede7d0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_done global_id 4297 payload 274 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3741273817 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8137330 msgr2=0x7f4ed8069330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8137330 0x7f4ed8069330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3741273817 >> v2:172.21.15.6:3300/0 conn(0x7f4ed8136f30 msgr2=0x7f4ed8068df0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:31.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 --2- 172.21.15.6:0/3741273817 >> v2:172.21.15.6:3300/0 conn(0x7f4ed8136f30 0x7f4ed8068df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3741273817 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed8096b80 con 0x7f4ed80986e0 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.779+0000 7f4ede7d0700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8137330 0x7f4ed8069330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3741273817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed80986e0 msgr2=0x7f4ed8098ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4eddfcf700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed80986e0 0x7f4ed8098ab0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ed809fe60 tx=0x7f4ed4009250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1ee87feb68aaddee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 1 -- 172.21.15.6:0/3741273817 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed400ba70 con 0x7f4ed80986e0 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 10 monclient: dump: 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.781+0000 7f4edcfcd700 1 -- 172.21.15.6:0/3741273817 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed400bbd0 con 0x7f4ed80986e0 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4edcfcd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4edcfcd700 1 -- 172.21.15.6:0/3741273817 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed4011420 con 0x7f4ed80986e0 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4edcfcd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4edcfcd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4edcfcd700 10 monclient: dump: 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.866 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4ee0233700 5 monclient: authenticate success, global_id 4297 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4ee0233700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.782+0000 7f4ee0233700 4 set_mon_vals no callback set 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 10 monclient: shutdown 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 -- 172.21.15.6:0/3741273817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed80986e0 msgr2=0x7f4ed8098ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed80986e0 0x7f4ed8098ab0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4ed809fe60 tx=0x7f4ed4009250 comp rx=0 tx=0).stop 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 -- 172.21.15.6:0/3741273817 shutdown_connections 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8137330 0x7f4ed8069330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3741273817 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed80986e0 0x7f4ed8098ab0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3741273817 >> v2:172.21.15.6:3300/0 conn(0x7f4ed8136f30 0x7f4ed8068df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 -- 172.21.15.6:0/3741273817 >> 172.21.15.6:0/3741273817 conn(0x7f4ed80a5480 msgr2=0x7f4ed80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 -- 172.21.15.6:0/3741273817 shutdown_connections 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.783+0000 7f4ee0233700 1 -- 172.21.15.6:0/3741273817 wait complete. 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.784+0000 7f4ee0233700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.784+0000 7f4ee0233700 10 monclient: monmap: 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:31.868 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.784+0000 7f4ee0233700 1 Processor -- start 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.785+0000 7f4ee0233700 1 -- start start 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.785+0000 7f4ee0233700 10 monclient: init 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.785+0000 7f4ee0233700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.785+0000 7f4ee0233700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 0x7f4ed806a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): picked mon.noname-a con 0x7f4ed80986e0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8136f30 0x7f4ed806ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): picked mon.noname-c con 0x7f4ed8136f30 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed8137330 0x7f4ed806b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): picked mon.noname-b con 0x7f4ed8137330 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed806e060 con 0x7f4ed80986e0 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed80a3b60 con 0x7f4ed8137330 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ed806cf00 con 0x7f4ed8136f30 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ee0233700 10 monclient(hunting): authenticate will time out at 1268.083108s 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 0x7f4ed806a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 0x7f4ed806a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42112/0 (socket says 172.21.15.6:42112) 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3537573045 learned_addr learned my addr 172.21.15.6:0/3537573045 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): get_auth_request con 0x7f4ed80986e0 auth_method 0 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.871 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed8137330 0x7f4ed806b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): get_auth_request con 0x7f4ed8137330 auth_method 0 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4ede7d0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient(hunting): handle_auth_done global_id 4290 payload 824 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8136f30 msgr2=0x7f4ed806ae50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T00:44:31.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8136f30 0x7f4ed806ae50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3537573045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed8137330 msgr2=0x7f4ed806b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed8137330 0x7f4ed806b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 monclient: found mon.noname-a 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.786+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3537573045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ed40094f0 con 0x7f4ed80986e0 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4eddfcf700 1 -- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 msgr2=0x7f4ed806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4eddfcf700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 0x7f4ed806a910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40090b0 tx=0x7f4ed4017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed401a660 con 0x7f4ed80986e0 2023-09-30T00:44:31.873 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: dump: 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.874 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ed401a7c0 con 0x7f4ed80986e0 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ed401ab40 con 0x7f4ed80986e0 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ec9ffb700 10 monclient: dump: 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.875 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ecbfff700 4 set_mon_vals no callback set 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 5 monclient: authenticate success, global_id 4290 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 10 monclient: _renew_subs 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ed806dd80 con 0x7f4ed80986e0 2023-09-30T00:44:31.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 10 monclient: _renew_subs 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.789+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ed806d950 con 0x7f4ed80986e0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.790+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4ed401b430 con 0x7f4ed80986e0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.791+0000 7f4ec9ffb700 1 --2- 172.21.15.6:0/3537573045 >> v2:172.21.15.22:6812/142603 conn(0x7f4ebc0357f0 0x7f4ebc037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.791+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4ed4048f20 con 0x7f4ed80986e0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.791+0000 7f4ee0233700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.791+0000 7f4ee0233700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.791+0000 7f4ee0233700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.791+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ed80a1340 con 0x7f4ed80986e0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.792+0000 7f4edd7ce700 1 --2- 172.21.15.6:0/3537573045 >> v2:172.21.15.22:6812/142603 conn(0x7f4ebc0357f0 0x7f4ebc037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.792+0000 7f4edd7ce700 10 monclient: get_auth_request con 0x7f4ebc0357f0 auth_method 0 2023-09-30T00:44:31.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.795+0000 7f4edd7ce700 1 --2- 172.21.15.6:0/3537573045 >> v2:172.21.15.22:6812/142603 conn(0x7f4ebc0357f0 0x7f4ebc037ca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009740 tx=0x7f4ecc006c50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.796+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4ed80a1340 con 0x7f4ed80986e0 2023-09-30T00:44:31.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.796+0000 7f4ec9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.796+0000 7f4ec9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.812+0000 7faa5b17c700 10 monclient: get_monmap_and_config 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.812+0000 7faa5b17c700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.812+0000 7faa5b17c700 10 monclient: monmap: 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:31.812510+0000 2023-09-30T00:44:31.881 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:31.812510+0000 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.813+0000 7faa5b17c700 1 Processor -- start 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.813+0000 7faa5b17c700 1 -- start start 2023-09-30T00:44:31.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.813+0000 7faa5b17c700 10 monclient: init 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7faa54098670 0x7faa54132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): picked mon.noname-a con 0x7faa54098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7faa54098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 0x7faa5412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): picked mon.noname-b con 0x7faa54098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa540a1000 con 0x7faa54098a40 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa540a0a50 con 0x7faa54098670 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa540a0730 con 0x7faa54098e90 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa5b17c700 10 monclient(hunting): authenticate will time out at 1268.111313s 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 0x7faa5412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 0x7faa5412f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36658/0 (socket says 172.21.15.6:36658) 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 1 -- 172.21.15.6:0/2515393944 learned_addr learned my addr 172.21.15.6:0/2515393944 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): get_auth_request con 0x7faa54098e90 auth_method 0 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.814+0000 7faa59719700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa58f18700 1 --2- 172.21.15.6:0/2515393944 >> v2:172.21.15.6:3300/0 conn(0x7faa54098670 0x7faa54132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa58f18700 10 monclient(hunting): get_auth_request con 0x7faa54098670 auth_method 0 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa58f18700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa58f18700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa58f18700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 10 monclient(hunting): handle_auth_done global_id 4292 payload 274 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 1 -- 172.21.15.6:0/2515393944 >> v2:172.21.15.6:3300/0 conn(0x7faa54098670 msgr2=0x7faa54132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 1 --2- 172.21.15.6:0/2515393944 >> v2:172.21.15.6:3300/0 conn(0x7faa54098670 0x7faa54132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 1 -- 172.21.15.6:0/2515393944 >> v1:172.21.15.6:6789/0 conn(0x7faa54098a40 legacy=0x7faa5412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 1 monclient: found mon.noname-b 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.815+0000 7faa59719700 1 -- 172.21.15.6:0/2515393944 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa54068df0 con 0x7faa54098e90 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.817+0000 7faa59719700 1 -- 172.21.15.6:0/2515393944 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 msgr2=0x7faa5412f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.817+0000 7faa59719700 1 --2- 172.21.15.6:0/2515393944 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 0x7faa5412f4f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009f20 tx=0x7faa4c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3a291157d3fac59 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 1 -- 172.21.15.6:0/2515393944 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c00e070 con 0x7faa54098e90 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: dump: 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.886 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 1 -- 172.21.15.6:0/2515393944 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa4c00bc50 con 0x7faa54098e90 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 1 -- 172.21.15.6:0/2515393944 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa4c0092e0 con 0x7faa54098e90 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa537fe700 10 monclient: dump: 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.888 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa5b17c700 5 monclient: authenticate success, global_id 4292 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa5b17c700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.818+0000 7faa5b17c700 4 set_mon_vals no callback set 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 10 monclient: shutdown 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 -- 172.21.15.6:0/2515393944 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 msgr2=0x7faa5412f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 --2- 172.21.15.6:0/2515393944 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 0x7faa5412f4f0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7faa4c009f20 tx=0x7faa4c009840 comp rx=0 tx=0).stop 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 -- 172.21.15.6:0/2515393944 shutdown_connections 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 --2- 172.21.15.6:0/2515393944 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098e90 0x7faa5412f4f0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 --2- 172.21.15.6:0/2515393944 >> v2:172.21.15.6:3300/0 conn(0x7faa54098670 0x7faa54132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 -- 172.21.15.6:0/2515393944 >> 172.21.15.6:0/2515393944 conn(0x7faa540a5480 msgr2=0x7faa540a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 -- 172.21.15.6:0/2515393944 shutdown_connections 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.819+0000 7faa5b17c700 1 -- 172.21.15.6:0/2515393944 wait complete. 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.820+0000 7faa5b17c700 10 monclient: build_initial_monmap 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.820+0000 7faa5b17c700 10 monclient: monmap: 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:31.890 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.820+0000 7faa5b17c700 1 Processor -- start 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.821+0000 7faa5b17c700 1 -- start start 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.821+0000 7faa5b17c700 10 monclient: init 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa54098670 0x7faa5414e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): picked mon.noname-a con 0x7faa54098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 0x7faa5414ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): picked mon.noname-c con 0x7faa54098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:31.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa54098e90 0x7faa5409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): picked mon.noname-b con 0x7faa54098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5409e4d0 con 0x7faa54098670 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa5409eb20 con 0x7faa54098e90 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa540a1500 con 0x7faa54098a40 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa5b17c700 10 monclient(hunting): authenticate will time out at 1268.119218s 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa54098670 0x7faa5414e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa54098670 0x7faa5414e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42140/0 (socket says 172.21.15.6:42140) 2023-09-30T00:44:31.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 1 -- 172.21.15.6:0/3183958510 learned_addr learned my addr 172.21.15.6:0/3183958510 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): get_auth_request con 0x7faa54098670 auth_method 0 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 0x7faa5414ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): get_auth_request con 0x7faa54098a40 auth_method 0 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa58f18700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.822+0000 7faa53fff700 10 monclient(hunting): handle_auth_done global_id 4295 payload 824 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 1 -- 172.21.15.6:0/3183958510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa54098e90 msgr2=0x7faa5409c7a0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa54098e90 0x7faa5409c7a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 1 -- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa54098670 msgr2=0x7faa5414e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa54098670 0x7faa5414e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:31.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 1 monclient: found mon.noname-c 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.823+0000 7faa53fff700 1 -- 172.21.15.6:0/3183958510 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa4c009ae0 con 0x7faa54098a40 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa53fff700 1 -- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 msgr2=0x7faa5414ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa53fff700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 0x7faa5414ec20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c510 tx=0x7faa44007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa440117a0 con 0x7faa54098a40 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: dump: 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.895 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa44011920 con 0x7faa54098a40 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa44011ce0 con 0x7faa54098a40 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:31.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa50ff9700 10 monclient: dump: 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:31.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa52ffd700 4 set_mon_vals no callback set 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa5b17c700 5 monclient: authenticate success, global_id 4295 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa5b17c700 10 monclient: _renew_subs 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa5b17c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.825+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa540a3af0 con 0x7faa54098a40 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.826+0000 7faa5b17c700 10 monclient: _renew_subs 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.826+0000 7faa5b17c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.826+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa5409df60 con 0x7faa54098a40 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7faa44012630 con 0x7faa54098a40 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa50ff9700 1 --2- 172.21.15.6:0/3183958510 >> v2:172.21.15.22:6812/142603 conn(0x7faa400357f0 0x7faa40037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7faa44002090 con 0x7faa54098a40 2023-09-30T00:44:31.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa5b17c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa5b17c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa5b17c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa54072b00 con 0x7faa54098a40 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa58f18700 1 --2- 172.21.15.6:0/3183958510 >> v2:172.21.15.22:6812/142603 conn(0x7faa400357f0 0x7faa40037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.827+0000 7faa58f18700 10 monclient: get_auth_request con 0x7faa400357f0 auth_method 0 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.830+0000 7faa58f18700 1 --2- 172.21.15.6:0/3183958510 >> v2:172.21.15.22:6812/142603 conn(0x7faa400357f0 0x7faa40037ca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa48006fd0 tx=0x7faa48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.832+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7faa4401c460 con 0x7faa54098a40 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.832+0000 7faa50ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:31.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.832+0000 7faa50ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:31.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.927+0000 7f3015e7e700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.927+0000 7f3015e7e700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.927+0000 7f3015e7e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.927+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f2ff0005630 con 0x7f3010098a40 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.928+0000 7f2ff77fe700 1 -- 172.21.15.6:0/458637215 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3004021030 con 0x7f3010098a40 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.928+0000 7f2ff77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.928+0000 7f2ff77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:31.928 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 >> v2:172.21.15.22:6812/142603 conn(0x7f2ffc0357a0 msgr2=0x7f2ffc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 --2- 172.21.15.6:0/458637215 >> v2:172.21.15.22:6812/142603 conn(0x7f2ffc0357a0 0x7f2ffc037c50 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f30000093a0 tx=0x7f3000016000 comp rx=0 tx=0).stop 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 10 monclient: shutdown 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 msgr2=0x7f301009d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 0x7f301009d570 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f300400ea50 tx=0x7f30040091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 shutdown_connections 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 --2- 172.21.15.6:0/458637215 >> v2:172.21.15.22:6812/142603 conn(0x7f2ffc0357a0 0x7f2ffc037c50 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3010098e90 0x7f301009dab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3010098670 0x7f301009d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 --2- 172.21.15.6:0/458637215 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3010098a40 0x7f301009d570 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 >> 172.21.15.6:0/458637215 conn(0x7f30100a5480 msgr2=0x7f30101309b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 shutdown_connections 2023-09-30T00:44:31.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.930+0000 7f3015e7e700 1 -- 172.21.15.6:0/458637215 wait complete. 2023-09-30T00:44:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.943+0000 7f0fc9947700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:31.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.943+0000 7f0fc9947700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.943+0000 7f0fc9947700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:31.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.943+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f0fc40982d0 con 0x7f0fc4098a40 2023-09-30T00:44:31.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.0 2023-09-30T00:44:31.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.944+0000 7f0faaffd700 1 -- 172.21.15.6:0/944242941 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0fb801a3d0 con 0x7f0fc4098a40 2023-09-30T00:44:31.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.944+0000 7f0faaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:31.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.944+0000 7f0faaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:31.947 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:31.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.947+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 >> v2:172.21.15.22:6812/142603 conn(0x7f0fb00357a0 msgr2=0x7f0fb0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.947+0000 7f0fc9947700 1 --2- 172.21.15.6:0/944242941 >> v2:172.21.15.22:6812/142603 conn(0x7f0fb00357a0 0x7f0fb0037c50 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4006f00 tx=0x7f0fb4008040 comp rx=0 tx=0).stop 2023-09-30T00:44:31.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.947+0000 7f0fc9947700 10 monclient: shutdown 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.947+0000 7f0fc9947700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.947+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 msgr2=0x7f0fc41419e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.947+0000 7f0fc9947700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 0x7f0fc41419e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0fc406a6b0 tx=0x7f0fb8016000 comp rx=0 tx=0).stop 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.948+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 shutdown_connections 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.948+0000 7f0fc9947700 1 --2- 172.21.15.6:0/944242941 >> v2:172.21.15.22:6812/142603 conn(0x7f0fb00357a0 0x7f0fb0037c50 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.948+0000 7f0fc9947700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0fc4098e90 0x7f0fc4141f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.948+0000 7f0fc9947700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0fc4098670 0x7f0fc409f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.948+0000 7f0fc9947700 1 --2- 172.21.15.6:0/944242941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0fc4098a40 0x7f0fc41419e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.948+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 >> 172.21.15.6:0/944242941 conn(0x7f0fc40a5480 msgr2=0x7f0fc4133c70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.949+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 shutdown_connections 2023-09-30T00:44:31.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:31.949+0000 7f0fc9947700 1 -- 172.21.15.6:0/944242941 wait complete. 2023-09-30T00:44:31.967 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.5 2023-09-30T00:44:32.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ee0233700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:32.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ee0233700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:32.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ee0233700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:32.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f4ed80a1340 con 0x7f4ed80986e0 2023-09-30T00:44:32.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ec9ffb700 1 -- 172.21.15.6:0/3537573045 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f4ed80a1340 con 0x7f4ed80986e0 2023-09-30T00:44:32.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ec9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:32.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.066+0000 7f4ec9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:32.067 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 >> v2:172.21.15.22:6812/142603 conn(0x7f4ebc0357f0 msgr2=0x7f4ebc037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3537573045 >> v2:172.21.15.22:6812/142603 conn(0x7f4ebc0357f0 0x7f4ebc037ca0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4ecc009740 tx=0x7f4ecc006c50 comp rx=0 tx=0).stop 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 10 monclient: shutdown 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 msgr2=0x7f4ed806a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 0x7f4ed806a910 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4ed40090b0 tx=0x7f4ed4017000 comp rx=0 tx=0).stop 2023-09-30T00:44:32.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 shutdown_connections 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3537573045 >> v2:172.21.15.22:6812/142603 conn(0x7f4ebc0357f0 0x7f4ebc037ca0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4ed8137330 0x7f4ed806b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4ed80986e0 0x7f4ed806a910 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 --2- 172.21.15.6:0/3537573045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4ed8136f30 0x7f4ed806ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 >> 172.21.15.6:0/3537573045 conn(0x7f4ed80a5480 msgr2=0x7f4ed8068da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 shutdown_connections 2023-09-30T00:44:32.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.069+0000 7f4ee0233700 1 -- 172.21.15.6:0/3537573045 wait complete. 2023-09-30T00:44:32.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.081+0000 7f3715931700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:32.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.081+0000 7f3715931700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:32.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.081+0000 7f3715931700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:32.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.081+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f36f0005360 con 0x7f3710098e90 2023-09-30T00:44:32.082 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.082+0000 7f36f6ffd700 1 -- 172.21.15.6:0/2133406804 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3708018690 con 0x7f3710098e90 2023-09-30T00:44:32.082 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.082+0000 7f36f6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:32.082 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.082+0000 7f36f6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:32.082 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:32.083 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.3 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 >> v2:172.21.15.22:6812/142603 conn(0x7f36fc0357f0 msgr2=0x7f36fc037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 --2- 172.21.15.6:0/2133406804 >> v2:172.21.15.22:6812/142603 conn(0x7f36fc0357f0 0x7f36fc037ca0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f37040093a0 tx=0x7f3704016000 comp rx=0 tx=0).stop 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 10 monclient: shutdown 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 msgr2=0x7f371009a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 0x7f371009a3c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f370800c930 tx=0x7f37080026e0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 shutdown_connections 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 --2- 172.21.15.6:0/2133406804 >> v2:172.21.15.22:6812/142603 conn(0x7f36fc0357f0 0x7f36fc037ca0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3710098e90 0x7f371009a3c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3710098670 0x7f3710099940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 --2- 172.21.15.6:0/2133406804 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3710098a40 0x7f3710099e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.084+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 >> 172.21.15.6:0/2133406804 conn(0x7f37100a5480 msgr2=0x7f371012f490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.085+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 shutdown_connections 2023-09-30T00:44:32.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.085+0000 7f3715931700 1 -- 172.21.15.6:0/2133406804 wait complete. 2023-09-30T00:44:32.097 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.1 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.100+0000 7f5220429700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.101+0000 7f5220429700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.101+0000 7f5220429700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.101+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f521806b220 con 0x7f5218132d00 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.101+0000 7f520e7fc700 1 -- 172.21.15.6:0/929499478 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f521806b220 con 0x7f5218132d00 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.101+0000 7f520e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:32.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.101+0000 7f520e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:32.102 INFO:teuthology.orchestra.run.smithi006.stdout:47244640258 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.103+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 >> v2:172.21.15.22:6812/142603 conn(0x7f51fc0357f0 msgr2=0x7f51fc037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.103+0000 7f5220429700 1 --2- 172.21.15.6:0/929499478 >> v2:172.21.15.22:6812/142603 conn(0x7f51fc0357f0 0x7f51fc037ca0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f5208007880 tx=0x7f5208004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 10 monclient: shutdown 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 msgr2=0x7f521812f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 0x7f521812f890 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f521400c510 tx=0x7f5214007ea0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 shutdown_connections 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 --2- 172.21.15.6:0/929499478 >> v2:172.21.15.22:6812/142603 conn(0x7f51fc0357f0 0x7f51fc037ca0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5218138d40 0x7f521806d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5218098670 0x7f521806e5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 --2- 172.21.15.6:0/929499478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5218132d00 0x7f521812f890 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 >> 172.21.15.6:0/929499478 conn(0x7f52180a5480 msgr2=0x7f52180a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 shutdown_connections 2023-09-30T00:44:32.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.104+0000 7f5220429700 1 -- 172.21.15.6:0/929499478 wait complete. 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa5b17c700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa5b17c700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa5b17c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7faa54063920 con 0x7faa54098a40 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa50ff9700 1 -- 172.21.15.6:0/3183958510 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7faa440102e0 con 0x7faa54098a40 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa50ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:32.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.107+0000 7faa50ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:32.108 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 >> v2:172.21.15.22:6812/142603 conn(0x7faa400357f0 msgr2=0x7faa40037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 --2- 172.21.15.6:0/3183958510 >> v2:172.21.15.22:6812/142603 conn(0x7faa400357f0 0x7faa40037ca0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faa48006fd0 tx=0x7faa48008040 comp rx=0 tx=0).stop 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 10 monclient: shutdown 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 msgr2=0x7faa5414ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 0x7faa5414ec20 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7faa4400c510 tx=0x7faa44007ea0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 shutdown_connections 2023-09-30T00:44:32.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 --2- 172.21.15.6:0/3183958510 >> v2:172.21.15.22:6812/142603 conn(0x7faa400357f0 0x7faa40037ca0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7faa54098e90 0x7faa5409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7faa54098670 0x7faa5414e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 --2- 172.21.15.6:0/3183958510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7faa54098a40 0x7faa5414ec20 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 >> 172.21.15.6:0/3183958510 conn(0x7faa540a5480 msgr2=0x7faa54069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 shutdown_connections 2023-09-30T00:44:32.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:32.110+0000 7faa5b17c700 1 -- 172.21.15.6:0/3183958510 wait complete. 2023-09-30T00:44:32.118 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640258 for osd.4 2023-09-30T00:44:32.123 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640259 for osd.2 2023-09-30T00:44:32.948 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2023-09-30T00:44:32.968 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2023-09-30T00:44:33.084 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.093+0000 7fd126e7f700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.093+0000 7fd126e7f700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.093+0000 7fd126e7f700 10 monclient: monmap: 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.093521+0000 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.093521+0000 2023-09-30T00:44:33.093 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.094 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 Processor -- start 2023-09-30T00:44:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 -- start start 2023-09-30T00:44:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient: init 2023-09-30T00:44:33.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098670 0x7fd120132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): picked mon.noname-b con 0x7fd120098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd120098a40 0x7fd12012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): picked mon.noname-a con 0x7fd120098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd120098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1200a1000 con 0x7fd120098e90 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1200a0a50 con 0x7fd120098a40 2023-09-30T00:44:33.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1200a0730 con 0x7fd120098670 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd126e7f700 10 monclient(hunting): authenticate will time out at 1269.391792s 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd124c1b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098670 0x7fd120132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd12541c700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd120098e90 0x7fd12012f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60936/0 (socket says 172.21.15.6:60936) 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd12541c700 1 -- 172.21.15.6:0/762346847 learned_addr learned my addr 172.21.15.6:0/762346847 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.094+0000 7fd11ffff700 1 --2- 172.21.15.6:0/762346847 >> v2:172.21.15.6:3300/0 conn(0x7fd120098a40 0x7fd12012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): get_auth_request con 0x7fd120098670 auth_method 0 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): get_auth_request con 0x7fd120098a40 auth_method 0 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient(hunting): handle_auth_done global_id 4299 payload 274 2023-09-30T00:44:33.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 1 -- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098670 msgr2=0x7fd120132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 1 --2- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098670 0x7fd120132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 1 -- 172.21.15.6:0/762346847 >> v1:172.21.15.6:6789/0 conn(0x7fd120098e90 legacy=0x7fd12012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 1 monclient: found mon.noname-a 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd11ffff700 1 -- 172.21.15.6:0/762346847 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd120068df0 con 0x7fd120098a40 2023-09-30T00:44:33.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.095+0000 7fd124c1b700 1 --2- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098670 0x7fd120132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:33.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11ffff700 1 -- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 msgr2=0x7fd12012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11ffff700 1 --2- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd1140098e0 tx=0x7fd1140091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8798f30276c5200 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 1 -- 172.21.15.6:0/762346847 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114011840 con 0x7fd120098a40 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: dump: 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.107 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd126e7f700 5 monclient: authenticate success, global_id 4299 2023-09-30T00:44:33.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 1 -- 172.21.15.6:0/762346847 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1140119a0 con 0x7fd120098a40 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd126e7f700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd126e7f700 4 set_mon_vals no callback set 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 1 -- 172.21.15.6:0/762346847 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114011d40 con 0x7fd120098a40 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.097+0000 7fd11f7fe700 10 monclient: dump: 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.109 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 10 monclient: shutdown 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 1 -- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 msgr2=0x7fd12012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 1 --2- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012ede0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fd1140098e0 tx=0x7fd1140091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 1 -- 172.21.15.6:0/762346847 shutdown_connections 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 1 --2- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098670 0x7fd120132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 1 --2- 172.21.15.6:0/762346847 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012ede0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.098+0000 7fd126e7f700 1 -- 172.21.15.6:0/762346847 >> 172.21.15.6:0/762346847 conn(0x7fd1200a5480 msgr2=0x7fd1200a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.099+0000 7fd126e7f700 1 -- 172.21.15.6:0/762346847 shutdown_connections 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.099+0000 7fd126e7f700 1 -- 172.21.15.6:0/762346847 wait complete. 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.099+0000 7fd126e7f700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.099+0000 7fd126e7f700 10 monclient: monmap: 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.111 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.099+0000 7fd126e7f700 1 Processor -- start 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 -- start start 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient: init 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd120141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.112 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): picked mon.noname-b con 0x7fd120098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): picked mon.noname-a con 0x7fd120098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd12013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): picked mon.noname-c con 0x7fd120098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd120131350 con 0x7fd120098a40 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12011c4a0 con 0x7fd120098670 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12011c780 con 0x7fd120098e90 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd126e7f700 10 monclient(hunting): authenticate will time out at 1269.397769s 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd12013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd11ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd12013e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36700/0 (socket says 172.21.15.6:36700) 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 1 -- 172.21.15.6:0/1701653080 learned_addr learned my addr 172.21.15.6:0/1701653080 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 10 monclient(hunting): get_auth_request con 0x7fd120098e90 auth_method 0 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd12541c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd11ffff700 10 monclient(hunting): get_auth_request con 0x7fd120098a40 auth_method 0 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.100+0000 7fd11ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd120141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): get_auth_request con 0x7fd120098670 auth_method 0 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd124c1b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient(hunting): handle_auth_done global_id 4302 payload 824 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 msgr2=0x7fd12013e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd12013e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 msgr2=0x7fd120141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd120141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 1 monclient: found mon.noname-a 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd110009420 con 0x7fd120098a40 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.101+0000 7fd12541c700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd12013e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:33.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 msgr2=0x7fd12012c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012c350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1200a1df0 tx=0x7fd114016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114000c40 con 0x7fd120098a40 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 10 monclient: dump: 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.117 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd126e7f700 5 monclient: authenticate success, global_id 4302 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd126e7f700 10 monclient: _renew_subs 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd126e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd114009420 con 0x7fd120098a40 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd114020070 con 0x7fd120098a40 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd11401b450 con 0x7fd120098a40 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.104+0000 7fd11cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.104+0000 7fd11cff9700 10 monclient: dump: 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.119 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.103+0000 7fd11effd700 4 set_mon_vals no callback set 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.104+0000 7fd126e7f700 10 monclient: _renew_subs 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.104+0000 7fd126e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.104+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12012f8d0 con 0x7fd120098a40 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.104+0000 7fd126e7f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd126e7f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd126e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 msgr2=0x7fd12012c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 msgr2=0x7fd12012c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T00:44:33.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12012e790 con 0x7fd120098a40 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012c350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1200a1df0 tx=0x7fd114016000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012c350 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fd1200a1df0 tx=0x7fd114016000 comp rx=0 tx=0).stop 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient: ms_handle_reset current mon [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 msgr2=0x7fd12012c350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd120098a40 0x7fd12012c350 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd10c005860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): picked mon.b con 0x7fd120098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 0x7fd10c0085e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.121 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): picked mon.a con 0x7fd10c006160 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd10c00b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): picked mon.c con 0x7fd120098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c00ba40 con 0x7fd10c006160 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c00bb80 con 0x7fd120098670 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10c00bcf0 con 0x7fd120098e90 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11cff9700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 0x7fd10c0085e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): get_auth_request con 0x7fd10c006160 auth_method 0 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd10c00b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): get_auth_request con 0x7fd120098e90 auth_method 0 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:44:33.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd12541c700 10 monclient(hunting): handle_auth_reply_more responding with 132 bytes 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd124c1b700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd10c005860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd124c1b700 10 monclient(hunting): get_auth_request con 0x7fd120098670 auth_method 0 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd124c1b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd124c1b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd124c1b700 10 monclient(hunting): _init_auth already have auth, reseting 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient(hunting): handle_auth_done global_id 4302 payload 293 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 msgr2=0x7fd10c00b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd10c00b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 msgr2=0x7fd10c005860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.124 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd10c005860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.125 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2023-09-30T00:44:33.126 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2023-09-30T00:44:33.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 monclient: found mon.a 2023-09-30T00:44:33.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fd10c00bfe0 con 0x7fd10c006160 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12012f8d0 con 0x7fd10c006160 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.105+0000 7fd11ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.108+0000 7fd11ffff700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 msgr2=0x7fd10c0085e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.108+0000 7fd11ffff700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 0x7fd10c0085e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1140093a0 tx=0x7fd114012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd114033040 con 0x7fd10c006160 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 10 monclient: dump: 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11401be20 con 0x7fd10c006160 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd114022030 con 0x7fd10c006160 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.109+0000 7fd11e7fc700 4 set_mon_vals no callback set 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.110+0000 7fd11cff9700 1 --2- 172.21.15.6:0/1701653080 >> v2:172.21.15.22:6812/142603 conn(0x7fd120098a40 0x7fd10c03fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.110+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 4 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd11404bcf0 con 0x7fd10c006160 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.110+0000 7fd124c1b700 1 --2- 172.21.15.6:0/1701653080 >> v2:172.21.15.22:6812/142603 conn(0x7fd120098a40 0x7fd10c03fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.110+0000 7fd124c1b700 10 monclient: get_auth_request con 0x7fd120098a40 auth_method 0 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.113+0000 7fd124c1b700 1 --2- 172.21.15.6:0/1701653080 >> v2:172.21.15.22:6812/142603 conn(0x7fd120098a40 0x7fd10c03fd90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1100090e0 tx=0x7fd110008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.115+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 5 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1140247e0 con 0x7fd10c006160 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.115+0000 7fd11cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.115+0000 7fd11cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:33.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.137+0000 7f8aaed12700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.137+0000 7f8aaed12700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.137+0000 7f8aaed12700 10 monclient: monmap: 2023-09-30T00:44:33.137 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.137 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.137560+0000 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.137560+0000 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 Processor -- start 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 -- start start 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient: init 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8aa8069580 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8aa809a8f0 0x7f8aa8136ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): picked mon.noname-c con 0x7f8aa809a8f0 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8aa8137430 0x7f8aa8139810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): picked mon.noname-a con 0x7f8aa8137430 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8aa8069dd0 con 0x7f8aa8069580 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa806a0b0 con 0x7f8aa8137430 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8aa806a390 con 0x7f8aa809a8f0 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.138+0000 7f8aaed12700 10 monclient(hunting): authenticate will time out at 1269.435962s 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8aa8137430 0x7f8aa8139810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8aa8137430 0x7f8aa8139810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42172/0 (socket says 172.21.15.6:42172) 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 -- 172.21.15.6:0/2037270093 learned_addr learned my addr 172.21.15.6:0/2037270093 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): get_auth_request con 0x7f8aa8137430 auth_method 0 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 1 --2- 172.21.15.6:0/2037270093 >> v2:172.21.15.22:3300/0 conn(0x7f8aa809a8f0 0x7f8aa8136ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): get_auth_request con 0x7f8aa809a8f0 auth_method 0 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient(hunting): handle_auth_done global_id 4305 payload 274 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 -- 172.21.15.6:0/2037270093 >> v2:172.21.15.22:3300/0 conn(0x7f8aa809a8f0 msgr2=0x7f8aa8136ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 --2- 172.21.15.6:0/2037270093 >> v2:172.21.15.22:3300/0 conn(0x7f8aa809a8f0 0x7f8aa8136ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 -- 172.21.15.6:0/2037270093 >> v1:172.21.15.22:6789/0 conn(0x7f8aa8069580 legacy=0x7f8aa809d100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 monclient: found mon.noname-a 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.139+0000 7f8aad2af700 1 -- 172.21.15.6:0/2037270093 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa8068bb0 con 0x7f8aa8137430 2023-09-30T00:44:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.141+0000 7f8aad2af700 1 -- 172.21.15.6:0/2037270093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8137430 msgr2=0x7f8aa8139810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.141+0000 7f8aad2af700 1 --2- 172.21.15.6:0/2037270093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8137430 0x7f8aa8139810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80685a0 tx=0x7f8aa0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94f37367226f751a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 1 -- 172.21.15.6:0/2037270093 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa000e070 con 0x7f8aa8137430 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: dump: 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.150 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 1 -- 172.21.15.6:0/2037270093 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8aa000bbf0 con 0x7f8aa8137430 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 1 -- 172.21.15.6:0/2037270093 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8aa00092e0 con 0x7f8aa8137430 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aa77fe700 10 monclient: dump: 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.151 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aaed12700 5 monclient: authenticate success, global_id 4305 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aaed12700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.142+0000 7f8aaed12700 4 set_mon_vals no callback set 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.143+0000 7f8aaed12700 10 monclient: shutdown 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.143+0000 7f8aaed12700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.143+0000 7f8aaed12700 1 -- 172.21.15.6:0/2037270093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8137430 msgr2=0x7f8aa8139810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.143+0000 7f8aaed12700 1 --2- 172.21.15.6:0/2037270093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8137430 0x7f8aa8139810 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f8aa80685a0 tx=0x7f8aa0009840 comp rx=0 tx=0).stop 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 1 -- 172.21.15.6:0/2037270093 shutdown_connections 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 1 --2- 172.21.15.6:0/2037270093 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8137430 0x7f8aa8139810 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 1 --2- 172.21.15.6:0/2037270093 >> v2:172.21.15.22:3300/0 conn(0x7f8aa809a8f0 0x7f8aa8136ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 1 -- 172.21.15.6:0/2037270093 >> 172.21.15.6:0/2037270093 conn(0x7f8aa80a32b0 msgr2=0x7f8aa80a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 1 -- 172.21.15.6:0/2037270093 shutdown_connections 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 1 -- 172.21.15.6:0/2037270093 wait complete. 2023-09-30T00:44:33.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.144+0000 7f8aaed12700 10 monclient: monmap: 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.145+0000 7f8aaed12700 1 Processor -- start 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.145+0000 7f8aaed12700 1 -- start start 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.145+0000 7f8aaed12700 10 monclient: init 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): picked mon.noname-a con 0x7f8aa8069580 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 0x7f8aa8136870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): picked mon.noname-b con 0x7f8aa809a8f0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8aa8137430 0x7f8aa814e190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): picked mon.noname-c con 0x7f8aa8137430 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa811bbf0 con 0x7f8aa8069580 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa8119ee0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8aa806f400 con 0x7f8aa8137430 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.146+0000 7f8aaed12700 10 monclient(hunting): authenticate will time out at 1269.443733s 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aad2af700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8aa8137430 0x7f8aa814e190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42188/0 (socket says 172.21.15.6:42188) 2023-09-30T00:44:33.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 -- 172.21.15.6:0/4257108657 learned_addr learned my addr 172.21.15.6:0/4257108657 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aad2af700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8aa8137430 0x7f8aa814e190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36718/0 (socket says 172.21.15.6:36718) 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 10 monclient(hunting): get_auth_request con 0x7f8aa8069580 auth_method 0 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 0x7f8aa8136870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): get_auth_request con 0x7f8aa809a8f0 auth_method 0 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aad2af700 10 monclient(hunting): get_auth_request con 0x7f8aa8137430 auth_method 0 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aad2af700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aad2af700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aad2af700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 -- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 msgr2=0x7f8aa814ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 -- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 msgr2=0x7f8aa814ae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aacaae700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient(hunting): handle_auth_done global_id 4309 payload 824 2023-09-30T00:44:33.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 -- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8aa8137430 msgr2=0x7f8aa814e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8aa8137430 0x7f8aa814e190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 -- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 msgr2=0x7f8aa814ae70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 monclient: found mon.noname-b 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.147+0000 7f8aa7fff700 1 -- 172.21.15.6:0/4257108657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa0009ae0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa7fff700 1 -- 172.21.15.6:0/4257108657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 msgr2=0x7f8aa8136870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa7fff700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 0x7f8aa8136870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400ea80 tx=0x7f8a940091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a940095d0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: dump: 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.161 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8a9400cad0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 5 monclient: authenticate success, global_id 4309 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 10 monclient: _renew_subs 2023-09-30T00:44:33.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aa80a28b0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400ce70 con 0x7f8aa809a8f0 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aa4ff9700 10 monclient: dump: 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.163 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 10 monclient: _renew_subs 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.151+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aa811aba0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.152+0000 7f8aa6ffd700 4 set_mon_vals no callback set 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8a94013680 con 0x7f8aa809a8f0 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aa4ff9700 1 --2- 172.21.15.6:0/4257108657 >> v2:172.21.15.22:6812/142603 conn(0x7f8a8c0357d0 0x7f8a8c037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8a94047ba0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aaed12700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aaed12700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aaed12700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a880050b0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.153+0000 7f8aacaae700 1 --2- 172.21.15.6:0/4257108657 >> v2:172.21.15.22:6812/142603 conn(0x7f8a8c0357d0 0x7f8a8c037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.154+0000 7f8aacaae700 10 monclient: get_auth_request con 0x7f8a8c0357d0 auth_method 0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.156+0000 7f8aacaae700 1 --2- 172.21.15.6:0/4257108657 >> v2:172.21.15.22:6812/142603 conn(0x7f8a8c0357d0 0x7f8a8c037c80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c006f20 tx=0x7f8a9c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.160+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8a940123d0 con 0x7f8aa809a8f0 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.160+0000 7f8aa4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.160+0000 7f8aa4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.273+0000 7f847dace700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.273+0000 7f847dace700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.273+0000 7f847dace700 10 monclient: monmap: 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.273553+0000 2023-09-30T00:44:33.273 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.273553+0000 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.274 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.274+0000 7f847dace700 1 Processor -- start 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.274+0000 7f847dace700 1 -- start start 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.274+0000 7f847dace700 10 monclient: init 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f847809d250 0x7f847809caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): picked mon.noname-a con 0x7f847809d250 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f847809dc90 0x7f847809e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): picked mon.noname-c con 0x7f847809dc90 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f847809e5d0 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:33.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f847809c3a0 con 0x7f847809e5d0 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847809c0c0 con 0x7f847809d250 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f847809bde0 con 0x7f847809dc90 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f847dace700 10 monclient(hunting): authenticate will time out at 1269.572655s 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f84777fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f847809d250 0x7f847809caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f84777fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f847809d250 0x7f847809caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42192/0 (socket says 172.21.15.6:42192) 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.275+0000 7f84777fe700 1 -- 172.21.15.6:0/1349176431 learned_addr learned my addr 172.21.15.6:0/1349176431 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): get_auth_request con 0x7f847809d250 auth_method 0 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 1 --2- 172.21.15.6:0/1349176431 >> v2:172.21.15.22:3300/0 conn(0x7f847809dc90 0x7f847809e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): get_auth_request con 0x7f847809dc90 auth_method 0 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f8476ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient(hunting): handle_auth_done global_id 4311 payload 274 2023-09-30T00:44:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 1 -- 172.21.15.6:0/1349176431 >> v2:172.21.15.22:3300/0 conn(0x7f847809dc90 msgr2=0x7f847809e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 1 --2- 172.21.15.6:0/1349176431 >> v2:172.21.15.22:3300/0 conn(0x7f847809dc90 0x7f847809e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 1 -- 172.21.15.6:0/1349176431 >> v1:172.21.15.22:6789/0 conn(0x7f847809e5d0 legacy=0x7f8478136b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 1 monclient: found mon.noname-a 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.276+0000 7f84777fe700 1 -- 172.21.15.6:0/1349176431 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847809abd0 con 0x7f847809d250 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.278+0000 7f84777fe700 1 -- 172.21.15.6:0/1349176431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809d250 msgr2=0x7f847809caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.278+0000 7f84777fe700 1 --2- 172.21.15.6:0/1349176431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809d250 0x7f847809caa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f847809a7c0 tx=0x7f846c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=659c23e545823faa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 1 -- 172.21.15.6:0/1349176431 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c009e70 con 0x7f847809d250 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: dump: 2023-09-30T00:44:33.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 1 -- 172.21.15.6:0/1349176431 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846c00ba30 con 0x7f847809d250 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f847dace700 5 monclient: authenticate success, global_id 4311 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f847dace700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 1 -- 172.21.15.6:0/1349176431 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846c00bdd0 con 0x7f847809d250 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f847dace700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f847dace700 4 set_mon_vals no callback set 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.279+0000 7f84767fc700 10 monclient: dump: 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.282 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.283 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.283 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 10 monclient: shutdown 2023-09-30T00:44:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 -- 172.21.15.6:0/1349176431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809d250 msgr2=0x7f847809caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 --2- 172.21.15.6:0/1349176431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809d250 0x7f847809caa0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f847809a7c0 tx=0x7f846c009250 comp rx=0 tx=0).stop 2023-09-30T00:44:33.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 -- 172.21.15.6:0/1349176431 shutdown_connections 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 --2- 172.21.15.6:0/1349176431 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809d250 0x7f847809caa0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 --2- 172.21.15.6:0/1349176431 >> v2:172.21.15.22:3300/0 conn(0x7f847809dc90 0x7f847809e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 -- 172.21.15.6:0/1349176431 >> 172.21.15.6:0/1349176431 conn(0x7f84780a3320 msgr2=0x7f84780a3720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 -- 172.21.15.6:0/1349176431 shutdown_connections 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.280+0000 7f847dace700 1 -- 172.21.15.6:0/1349176431 wait complete. 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.281+0000 7f847dace700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.281+0000 7f847dace700 10 monclient: monmap: 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.286 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.282+0000 7f847dace700 1 Processor -- start 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.282+0000 7f847dace700 1 -- start start 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.282+0000 7f847dace700 10 monclient: init 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:33.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f847809d250 0x7f8478142590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): picked mon.noname-b con 0x7f847809d250 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 0x7f8478141c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): picked mon.noname-c con 0x7f847809dc90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809e5d0 0x7f8478145ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): picked mon.noname-a con 0x7f847809e5d0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84781272a0 con 0x7f847809e5d0 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8478098f40 con 0x7f847809d250 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f847811bf80 con 0x7f847809dc90 2023-09-30T00:44:33.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.283+0000 7f847dace700 10 monclient(hunting): authenticate will time out at 1269.580703s 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 0x7f8478141c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 0x7f8478141c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36726/0 (socket says 172.21.15.6:36726) 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 -- 172.21.15.6:0/2813349789 learned_addr learned my addr 172.21.15.6:0/2813349789 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): get_auth_request con 0x7f847809dc90 auth_method 0 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f847809d250 0x7f8478142590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): get_auth_request con 0x7f847809d250 auth_method 0 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f84777fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809e5d0 0x7f8478145ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): get_auth_request con 0x7f847809e5d0 auth_method 0 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8477fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient(hunting): handle_auth_done global_id 4307 payload 824 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 -- 172.21.15.6:0/2813349789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f847809d250 msgr2=0x7f8478142590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f847809d250 0x7f8478142590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 -- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809e5d0 msgr2=0x7f8478145ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809e5d0 0x7f8478145ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 monclient: found mon.noname-c 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.284+0000 7f8476ffd700 1 -- 172.21.15.6:0/2813349789 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f846c0094f0 con 0x7f847809dc90 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f8476ffd700 1 -- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 msgr2=0x7f8478141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f8476ffd700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 0x7f8478141c20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f846800ec80 tx=0x7f84680091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f846800ca60 con 0x7f847809dc90 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: dump: 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.293 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f846800cbc0 con 0x7f847809dc90 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f847dace700 5 monclient: authenticate success, global_id 4307 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8468013470 con 0x7f847809dc90 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f845f7fe700 10 monclient: dump: 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.295 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f847dace700 10 monclient: _renew_subs 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f847dace700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8478126d00 con 0x7f847809dc90 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.287+0000 7f8475ffb700 4 set_mon_vals no callback set 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.289+0000 7f847dace700 10 monclient: _renew_subs 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.289+0000 7f847dace700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.289+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f847811d420 con 0x7f847809dc90 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.289+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f84680137e0 con 0x7f847809dc90 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.289+0000 7f845f7fe700 1 --2- 172.21.15.6:0/2813349789 >> v2:172.21.15.22:6812/142603 conn(0x7f84640357f0 0x7f8464037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.289+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8468011750 con 0x7f847809dc90 2023-09-30T00:44:33.296 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.290+0000 7f84777fe700 1 --2- 172.21.15.6:0/2813349789 >> v2:172.21.15.22:6812/142603 conn(0x7f84640357f0 0x7f8464037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.290+0000 7f84777fe700 10 monclient: get_auth_request con 0x7f84640357f0 auth_method 0 2023-09-30T00:44:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.290+0000 7f847dace700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.290+0000 7f847dace700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.290+0000 7f847dace700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.290+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8478072b00 con 0x7f847809dc90 2023-09-30T00:44:33.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.292+0000 7f84777fe700 1 --2- 172.21.15.6:0/2813349789 >> v2:172.21.15.22:6812/142603 conn(0x7f84640357f0 0x7f8464037ca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f847809a7c0 tx=0x7f846c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.300+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f846804d060 con 0x7f847809dc90 2023-09-30T00:44:33.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.300+0000 7f845f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.300+0000 7f845f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.302+0000 7fd29608d700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.302+0000 7fd29608d700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.302+0000 7fd29608d700 10 monclient: monmap: 2023-09-30T00:44:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.302 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.302777+0000 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.302777+0000 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.303 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fd29608d700 1 Processor -- start 2023-09-30T00:44:33.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fd29608d700 1 -- start start 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fd29608d700 10 monclient: init 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fbcc9907700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fbcc9907700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fbcc9907700 10 monclient: monmap: 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.305 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.305636+0000 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.305636+0000 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.306 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.305+0000 7fd29608d700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.307 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fd290098670 0x7fd290132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): picked mon.noname-c con 0x7fd290098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 0x7fd29012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): picked mon.noname-a con 0x7fd290098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): picked mon.noname-b con 0x7fd290098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2900a0a50 con 0x7fd290098a40 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2900a1000 con 0x7fd290098670 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.308 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900a0730 con 0x7fd290098e90 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd29608d700 10 monclient(hunting): authenticate will time out at 1269.603112s 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 0x7fd29012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 0x7fd29012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42214/0 (socket says 172.21.15.6:42214) 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 1 -- 172.21.15.6:0/409895734 learned_addr learned my addr 172.21.15.6:0/409895734 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): get_auth_request con 0x7fd290098a40 auth_method 0 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/409895734 >> v2:172.21.15.22:3300/0 conn(0x7fd290098670 0x7fd290132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): get_auth_request con 0x7fd290098670 auth_method 0 2023-09-30T00:44:33.309 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_done global_id 4318 payload 274 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28f7fe700 1 -- 172.21.15.6:0/409895734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 msgr2=0x7fd29012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.306+0000 7fd28ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/409895734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 1 -- 172.21.15.6:0/409895734 >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 msgr2=0x7fd29012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/409895734 >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 0x7fd29012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 1 monclient: found mon.noname-c 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28f7fe700 1 -- 172.21.15.6:0/409895734 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd290068df0 con 0x7fd290098670 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fbcc9907700 1 Processor -- start 2023-09-30T00:44:33.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fbcc9907700 1 -- start start 2023-09-30T00:44:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fbcc9907700 10 monclient: init 2023-09-30T00:44:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.307+0000 7fd28effd700 1 --2- 172.21.15.6:0/409895734 >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 0x7fd29012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:33.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient: _add_conns ranks=[0,3,4,2,1] 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbcc4098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fbcc4132d00 0x7fbcc4137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): picked mon.noname-a con 0x7fbcc4132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fbcc4137b60 0x7fbcc4139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): picked mon.noname-c con 0x7fbcc4137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbcc40a1000 con 0x7fbcc4098670 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc4099150 con 0x7fbcc4132d00 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbcc40a0a50 con 0x7fbcc4137b60 2023-09-30T00:44:33.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc9907700 10 monclient(hunting): authenticate will time out at 1269.605327s 2023-09-30T00:44:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc2ffd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbcc4098670 0x7fbcc4098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60940/0 (socket says 172.21.15.6:60940) 2023-09-30T00:44:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.308+0000 7fbcc2ffd700 1 -- 172.21.15.6:0/583052639 learned_addr learned my addr 172.21.15.6:0/583052639 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fd28ffff700 10 monclient: get_auth_request con 0x7fd290098e90 auth_method 0 2023-09-30T00:44:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fd28ffff700 1 --2- 172.21.15.6:0/409895734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:44:33.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 1 --2- 172.21.15.6:0/583052639 >> v2:172.21.15.22:3300/0 conn(0x7fbcc4137b60 0x7fbcc4139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fd28f7fe700 1 -- 172.21.15.6:0/409895734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098670 msgr2=0x7fd290132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/409895734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098670 0x7fd290132d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd29009fe60 tx=0x7fd2840091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=267506b900485b1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): get_auth_request con 0x7fbcc4137b60 auth_method 0 2023-09-30T00:44:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fd28e7fc700 1 -- 172.21.15.6:0/409895734 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284011840 con 0x7fd290098670 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fd28e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: dump: 2023-09-30T00:44:33.316 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 1 -- 172.21.15.6:0/409895734 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2840119a0 con 0x7fd290098670 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 1 -- 172.21.15.6:0/409895734 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd284011d60 con 0x7fd290098670 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd28e7fc700 10 monclient: dump: 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.318 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd29608d700 5 monclient: authenticate success, global_id 4318 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd29608d700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.310+0000 7fd29608d700 4 set_mon_vals no callback set 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_done global_id 4321 payload 274 2023-09-30T00:44:33.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 1 -- 172.21.15.6:0/583052639 >> v2:172.21.15.6:3300/0 conn(0x7fbcc4132d00 msgr2=0x7fbcc4137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 1 --2- 172.21.15.6:0/583052639 >> v2:172.21.15.6:3300/0 conn(0x7fbcc4132d00 0x7fbcc4137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 1 -- 172.21.15.6:0/583052639 >> v1:172.21.15.6:6789/0 conn(0x7fbcc4098670 legacy=0x7fbcc4098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 1 monclient: found mon.noname-c 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.309+0000 7fbcc37fe700 1 -- 172.21.15.6:0/583052639 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcc40a0730 con 0x7fbcc4137b60 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 10 monclient: shutdown 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 -- 172.21.15.6:0/409895734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098670 msgr2=0x7fd290132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 --2- 172.21.15.6:0/409895734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098670 0x7fd290132d00 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd29009fe60 tx=0x7fd2840091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 -- 172.21.15.6:0/409895734 shutdown_connections 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 --2- 172.21.15.6:0/409895734 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 --2- 172.21.15.6:0/409895734 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098670 0x7fd290132d00 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 --2- 172.21.15.6:0/409895734 >> v2:172.21.15.6:3300/0 conn(0x7fd290098a40 0x7fd29012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 -- 172.21.15.6:0/409895734 >> 172.21.15.6:0/409895734 conn(0x7fd2900a5480 msgr2=0x7fd2900a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 -- 172.21.15.6:0/409895734 shutdown_connections 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.311+0000 7fd29608d700 1 -- 172.21.15.6:0/409895734 wait complete. 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.312+0000 7fd29608d700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.312+0000 7fd29608d700 10 monclient: monmap: 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.322 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.323 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc37fe700 1 -- 172.21.15.6:0/583052639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4137b60 msgr2=0x7fbcc4139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc37fe700 1 --2- 172.21.15.6:0/583052639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4137b60 0x7fbcc4139f40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40a3330 tx=0x7fbcb80091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f394ef2e9f75baee server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fd29608d700 1 Processor -- start 2023-09-30T00:44:33.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc1ffb700 1 -- 172.21.15.6:0/583052639 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800b960 con 0x7fbcc4137b60 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fd29608d700 1 -- start start 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fd29608d700 10 monclient: init 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc1ffb700 10 monclient: dump: 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.324 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc9907700 5 monclient: authenticate success, global_id 4321 2023-09-30T00:44:33.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.313+0000 7fbcc9907700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:33.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd290098670 0x7fd29009eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): picked mon.noname-a con 0x7fd290098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098a40 0x7fd29012e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): picked mon.noname-b con 0x7fd290098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): picked mon.noname-c con 0x7fd290098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2900a1500 con 0x7fd290098670 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd29009e1a0 con 0x7fd290098a40 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd29009dec0 con 0x7fd290098e90 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd29608d700 10 monclient(hunting): authenticate will time out at 1269.611350s 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012ec80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36746/0 (socket says 172.21.15.6:36746) 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 1 -- 172.21.15.6:0/810488304 learned_addr learned my addr 172.21.15.6:0/810488304 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): get_auth_request con 0x7fd290098e90 auth_method 0 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd290098670 0x7fd29009eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): get_auth_request con 0x7fd290098670 auth_method 0 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098a40 0x7fd29012e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 10 monclient(hunting): get_auth_request con 0x7fd290098a40 auth_method 0 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.314+0000 7fd28effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 10 monclient(hunting): handle_auth_done global_id 4310 payload 824 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 1 -- 172.21.15.6:0/810488304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098a40 msgr2=0x7fd29012e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098a40 0x7fd29012e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 1 -- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd290098670 msgr2=0x7fd29009eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd290098670 0x7fd29009eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 1 monclient: found mon.noname-c 2023-09-30T00:44:33.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.315+0000 7fd28ffff700 1 -- 172.21.15.6:0/810488304 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2800094f0 con 0x7fd290098e90 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc1ffb700 1 -- 172.21.15.6:0/583052639 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcb800bac0 con 0x7fbcc4137b60 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc1ffb700 1 -- 172.21.15.6:0/583052639 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcb800be60 con 0x7fbcc4137b60 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc1ffb700 10 monclient: dump: 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc9907700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.316+0000 7fbcc9907700 4 set_mon_vals no callback set 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 10 monclient: shutdown 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 -- 172.21.15.6:0/583052639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4137b60 msgr2=0x7fbcc4139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 --2- 172.21.15.6:0/583052639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4137b60 0x7fbcc4139f40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fbcc40a3330 tx=0x7fbcb80091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 -- 172.21.15.6:0/583052639 shutdown_connections 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 --2- 172.21.15.6:0/583052639 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4137b60 0x7fbcc4139f40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 --2- 172.21.15.6:0/583052639 >> v2:172.21.15.6:3300/0 conn(0x7fbcc4132d00 0x7fbcc4137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 -- 172.21.15.6:0/583052639 >> 172.21.15.6:0/583052639 conn(0x7fbcc40a5480 msgr2=0x7fbcc40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 -- 172.21.15.6:0/583052639 shutdown_connections 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.317+0000 7fbcc9907700 1 -- 172.21.15.6:0/583052639 wait complete. 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.318+0000 7fbcc9907700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.318+0000 7fbcc9907700 10 monclient: monmap: 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.334 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.318+0000 7fbcc9907700 1 Processor -- start 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.318+0000 7fd28ffff700 1 -- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 msgr2=0x7fd29012ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.318+0000 7fd28ffff700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012ec80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd28800cfb0 tx=0x7fd2880078c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 -- start start 2023-09-30T00:44:33.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient: init 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28800aea0 con 0x7fd290098e90 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: dump: 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd28800d650 con 0x7fd290098e90 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd28800da10 con 0x7fd290098e90 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd2777fe700 10 monclient: dump: 2023-09-30T00:44:33.338 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 5 monclient: authenticate success, global_id 4310 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 10 monclient: _renew_subs 2023-09-30T00:44:33.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd284009420 con 0x7fd290098e90 2023-09-30T00:44:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 10 monclient: _renew_subs 2023-09-30T00:44:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd29009f410 con 0x7fd290098e90 2023-09-30T00:44:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fd28dffb700 4 set_mon_vals no callback set 2023-09-30T00:44:33.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 0x7fbcc4137400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): picked mon.noname-b con 0x7fbcc4098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbcc4132d00 0x7fbcc409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): picked mon.noname-c con 0x7fbcc4132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcc4137b60 0x7fbcc409cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): picked mon.noname-a con 0x7fbcc4137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc4133ad0 con 0x7fbcc4137b60 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc40a1500 con 0x7fbcc4098670 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbcc4069ff0 con 0x7fbcc4132d00 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.319+0000 7fbcc9907700 10 monclient(hunting): authenticate will time out at 1269.616994s 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbcc4132d00 0x7fbcc409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbcc4132d00 0x7fbcc409c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36750/0 (socket says 172.21.15.6:36750) 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 1 -- 172.21.15.6:0/843218204 learned_addr learned my addr 172.21.15.6:0/843218204 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): get_auth_request con 0x7fbcc4132d00 auth_method 0 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcc4137b60 0x7fbcc409cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): get_auth_request con 0x7fbcc4137b60 auth_method 0 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc2ffd700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 0x7fbcc4137400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc2ffd700 10 monclient(hunting): get_auth_request con 0x7fbcc4098670 auth_method 0 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.320+0000 7fbcc27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 10 monclient(hunting): handle_auth_done global_id 4327 payload 824 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 1 -- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbcc4132d00 msgr2=0x7fbcc409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbcc4132d00 0x7fbcc409c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 1 -- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcc4137b60 msgr2=0x7fbcc409cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcc4137b60 0x7fbcc409cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 1 monclient: found mon.noname-b 2023-09-30T00:44:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc2ffd700 1 -- 172.21.15.6:0/843218204 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbcb8009420 con 0x7fbcc4098670 2023-09-30T00:44:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fbcc37fe700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcc4137b60 0x7fbcc409cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:33.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.321+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd288003020 con 0x7fd290098e90 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd29608d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd29608d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd29608d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2700050b0 con 0x7fd290098e90 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd2777fe700 1 --2- 172.21.15.6:0/810488304 >> v2:172.21.15.22:6812/142603 conn(0x7fd27c035820 0x7fd27c037cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd288047f70 con 0x7fd290098e90 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/810488304 >> v2:172.21.15.22:6812/142603 conn(0x7fd27c035820 0x7fd27c037cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.322+0000 7fd28f7fe700 10 monclient: get_auth_request con 0x7fd27c035820 auth_method 0 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.324+0000 7fd966fe9700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.324+0000 7fd966fe9700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.324+0000 7fd966fe9700 10 monclient: monmap: 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.324913+0000 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.324913+0000 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.347 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.348 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.348 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.325+0000 7fd28f7fe700 1 --2- 172.21.15.6:0/810488304 >> v2:172.21.15.22:6812/142603 conn(0x7fd27c035820 0x7fd27c037cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd2840093f0 tx=0x7fd284017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.325+0000 7fd966fe9700 1 Processor -- start 2023-09-30T00:44:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.325+0000 7fd966fe9700 1 -- start start 2023-09-30T00:44:33.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.325+0000 7fd966fe9700 10 monclient: init 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient: _add_conns ranks=[0,1,2,3,4] 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd96009cd40 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd96009a8f0 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013edf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient(hunting): picked mon.noname-b con 0x7fd960069730 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd960068820 con 0x7fd96009cd40 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd96009f090 con 0x7fd96009a8f0 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96006a0b0 con 0x7fd960069730 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.326+0000 7fd966fe9700 10 monclient(hunting): authenticate will time out at 1269.623834s 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd964d85700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd96009cd40 0x7fd96009d110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60942/0 (socket says 172.21.15.6:60942) 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd964d85700 1 -- 172.21.15.6:0/2594481085 learned_addr learned my addr 172.21.15.6:0/2594481085 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd95ffff700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fd96009a8f0 0x7fd96013b6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:57398/0 (socket says 172.21.15.6:57398) 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 1 --2- 172.21.15.6:0/2594481085 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): get_auth_request con 0x7fd960069730 auth_method 0 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient(hunting): handle_auth_done global_id 4316 payload 274 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 1 -- 172.21.15.6:0/2594481085 >> v1:172.21.15.22:6789/0 conn(0x7fd96009a8f0 legacy=0x7fd96013b6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 1 -- 172.21.15.6:0/2594481085 >> v1:172.21.15.6:6789/0 conn(0x7fd96009cd40 legacy=0x7fd96009d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 1 monclient: found mon.noname-b 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.327+0000 7fd965586700 1 -- 172.21.15.6:0/2594481085 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd96006a6e0 con 0x7fd960069730 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.330+0000 7fd965586700 1 -- 172.21.15.6:0/2594481085 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 msgr2=0x7fd96013edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.330+0000 7fd965586700 1 --2- 172.21.15.6:0/2594481085 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013edf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd958009e90 tx=0x7fd958009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8f9fb656be7b575e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc2ffd700 1 -- 172.21.15.6:0/843218204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 msgr2=0x7fbcc4137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc2ffd700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 0x7fbcc4137400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc01dce0 tx=0x7fbcbc005450 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc01bab0 con 0x7fbcc4098670 2023-09-30T00:44:33.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: dump: 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.354 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbcbc01bc30 con 0x7fbcc4098670 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 5 monclient: authenticate success, global_id 4327 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 10 monclient: _renew_subs 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbcc4134160 con 0x7fbcc4098670 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcbc0070f0 con 0x7fbcc4098670 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:44:33.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcaaffd700 10 monclient: dump: 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 10 monclient: _renew_subs 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbcc409e550 con 0x7fbcc4098670 2023-09-30T00:44:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.331+0000 7fbcc17fa700 4 set_mon_vals no callback set 2023-09-30T00:44:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd288046e60 con 0x7fd290098e90 2023-09-30T00:44:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd2777fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd2777fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 1 -- 172.21.15.6:0/2594481085 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800e070 con 0x7fd960069730 2023-09-30T00:44:33.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: dump: 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.358 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 1 -- 172.21.15.6:0/2594481085 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd958012430 con 0x7fd960069730 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 1 -- 172.21.15.6:0/2594481085 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd95800bea0 con 0x7fd960069730 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd95f7fe700 10 monclient: dump: 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.359 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbcbc0074c0 con 0x7fbcc4098670 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fbcaaffd700 1 --2- 172.21.15.6:0/843218204 >> v2:172.21.15.22:6812/142603 conn(0x7fbcb00357f0 0x7fbcb0037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbcbc058c90 con 0x7fbcc4098670 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fbcc9907700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fbcc9907700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fbcc9907700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbcc4133930 con 0x7fbcc4098670 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fbcc27fc700 1 --2- 172.21.15.6:0/843218204 >> v2:172.21.15.22:6812/142603 conn(0x7fbcb00357f0 0x7fbcb0037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fbcc27fc700 10 monclient: get_auth_request con 0x7fbcb00357f0 auth_method 0 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd966fe9700 5 monclient: authenticate success, global_id 4316 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd966fe9700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.332+0000 7fd966fe9700 4 set_mon_vals no callback set 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fd966fe9700 10 monclient: shutdown 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fd966fe9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fd966fe9700 1 -- 172.21.15.6:0/2594481085 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 msgr2=0x7fd96013edf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.333+0000 7fd966fe9700 1 --2- 172.21.15.6:0/2594481085 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013edf0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fd958009e90 tx=0x7fd958009780 comp rx=0 tx=0).stop 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.334+0000 7fd966fe9700 1 -- 172.21.15.6:0/2594481085 shutdown_connections 2023-09-30T00:44:33.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.334+0000 7fd966fe9700 1 --2- 172.21.15.6:0/2594481085 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013edf0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.334+0000 7fd966fe9700 1 -- 172.21.15.6:0/2594481085 >> 172.21.15.6:0/2594481085 conn(0x7fd9600a32b0 msgr2=0x7fd9600a36b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.335+0000 7fd966fe9700 1 -- 172.21.15.6:0/2594481085 shutdown_connections 2023-09-30T00:44:33.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.336+0000 7fbcc27fc700 1 --2- 172.21.15.6:0/843218204 >> v2:172.21.15.22:6812/142603 conn(0x7fbcb00357f0 0x7fbcb0037ca0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4009ca0 tx=0x7fbcb4009250 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.335+0000 7fd966fe9700 1 -- 172.21.15.6:0/2594481085 wait complete. 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.335+0000 7fd966fe9700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.335+0000 7fd966fe9700 10 monclient: monmap: 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.365 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.336+0000 7fd966fe9700 1 Processor -- start 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.336+0000 7fd966fe9700 1 -- start start 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.336+0000 7fd966fe9700 10 monclient: init 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): picked mon.noname-c con 0x7fd960069730 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd96009a8f0 0x7fd96011a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): picked mon.noname-b con 0x7fd96009a8f0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd96009cd40 0x7fd96011acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): picked mon.noname-a con 0x7fd96009cd40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96006f1d0 con 0x7fd96009cd40 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd96009c360 con 0x7fd96009a8f0 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd96011d010 con 0x7fd960069730 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd966fe9700 10 monclient(hunting): authenticate will time out at 1269.634632s 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36772/0 (socket says 172.21.15.6:36772) 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 1 -- 172.21.15.6:0/1774389629 learned_addr learned my addr 172.21.15.6:0/1774389629 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 10 monclient(hunting): get_auth_request con 0x7fd960069730 auth_method 0 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.337+0000 7fd964d85700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 10 monclient(hunting): handle_auth_done global_id 4319 payload 824 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 1 -- 172.21.15.6:0/1774389629 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd96009a8f0 msgr2=0x7fd96011a770 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd96009a8f0 0x7fd96011a770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 1 -- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd96009cd40 msgr2=0x7fd96011acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd96009cd40 0x7fd96011acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 1 monclient: found mon.noname-c 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fd964d85700 1 -- 172.21.15.6:0/1774389629 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd958009a20 con 0x7fd960069730 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbcc4133930 con 0x7fbcc4098670 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fbcaaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.338+0000 7fbcaaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:33.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd964d85700 1 -- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 msgr2=0x7fd96013ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd964d85700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013ed30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c5d0 tx=0x7fd954007f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954007610 con 0x7fd960069730 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: dump: 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9540057f0 con 0x7fd960069730 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd954005bb0 con 0x7fd960069730 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:33.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95cff9700 10 monclient: dump: 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.342+0000 7fd95effd700 4 set_mon_vals no callback set 2023-09-30T00:44:33.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 5 monclient: authenticate success, global_id 4319 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 10 monclient: _renew_subs 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd96013b040 con 0x7fd960069730 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 10 monclient: _renew_subs 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.346+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd96013e160 con 0x7fd960069730 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd954014420 con 0x7fd960069730 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd966fe9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd966fe9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd966fe9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd960072b00 con 0x7fd960069730 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd95cff9700 1 --2- 172.21.15.6:0/1774389629 >> v2:172.21.15.22:6812/142603 conn(0x7fd944035820 0x7fd944037cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.347+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd954047880 con 0x7fd960069730 2023-09-30T00:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.348+0000 7fd95ffff700 1 --2- 172.21.15.6:0/1774389629 >> v2:172.21.15.22:6812/142603 conn(0x7fd944035820 0x7fd944037cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.348+0000 7fd95ffff700 10 monclient: get_auth_request con 0x7fd944035820 auth_method 0 2023-09-30T00:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.350+0000 7fd95ffff700 1 --2- 172.21.15.6:0/1774389629 >> v2:172.21.15.22:6812/142603 conn(0x7fd944035820 0x7fd944037cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd96011c6b0 tx=0x7fd94c007b50 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.357+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9540108b0 con 0x7fd960069730 2023-09-30T00:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.357+0000 7fd95cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.357+0000 7fd95cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.372+0000 7fd126e7f700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.372+0000 7fd126e7f700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.372+0000 7fd126e7f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.372+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fd12012f8d0 con 0x7fd10c006160 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.373+0000 7fd11cff9700 1 -- 172.21.15.6:0/1701653080 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd11403d070 con 0x7fd10c006160 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.373+0000 7fd11cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2023-09-30T00:44:33.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.373+0000 7fd11cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:33.378 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:33.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.375+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 >> v2:172.21.15.22:6812/142603 conn(0x7fd120098a40 msgr2=0x7fd10c03fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.375+0000 7fd126e7f700 1 --2- 172.21.15.6:0/1701653080 >> v2:172.21.15.22:6812/142603 conn(0x7fd120098a40 0x7fd10c03fd90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fd1100090e0 tx=0x7fd110008040 comp rx=0 tx=0).stop 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.375+0000 7fd126e7f700 10 monclient: shutdown 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.375+0000 7fd126e7f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.375+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 msgr2=0x7fd10c0085e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.375+0000 7fd126e7f700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 0x7fd10c0085e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd1140093a0 tx=0x7fd114012000 comp rx=0 tx=0).stop 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.376+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 shutdown_connections 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.376+0000 7fd126e7f700 1 --2- 172.21.15.6:0/1701653080 >> v2:172.21.15.22:6812/142603 conn(0x7fd120098a40 0x7fd10c03fd90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.376+0000 7fd126e7f700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd120098e90 0x7fd10c00b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.376+0000 7fd126e7f700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd10c006160 0x7fd10c0085e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.376+0000 7fd126e7f700 1 --2- 172.21.15.6:0/1701653080 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd120098670 0x7fd10c005860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.376+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 >> 172.21.15.6:0/1701653080 conn(0x7fd1200a5480 msgr2=0x7fd12012fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.380+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 shutdown_connections 2023-09-30T00:44:33.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.380+0000 7fd126e7f700 1 -- 172.21.15.6:0/1701653080 wait complete. 2023-09-30T00:44:33.394 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.0 2023-09-30T00:44:33.394 DEBUG:teuthology.parallel:result is None 2023-09-30T00:44:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.557+0000 7f8aaed12700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.557+0000 7f8aaed12700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.557+0000 7f8aaed12700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.557+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f8a88005360 con 0x7f8aa809a8f0 2023-09-30T00:44:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.558+0000 7f8aa4ff9700 1 -- 172.21.15.6:0/4257108657 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8a94011350 con 0x7f8aa809a8f0 2023-09-30T00:44:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.558+0000 7f8aa4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2023-09-30T00:44:33.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.558+0000 7f8aa4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:33.558 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:33.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.561+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 >> v2:172.21.15.22:6812/142603 conn(0x7f8a8c0357d0 msgr2=0x7f8a8c037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.561+0000 7f8aaed12700 1 --2- 172.21.15.6:0/4257108657 >> v2:172.21.15.22:6812/142603 conn(0x7f8a8c0357d0 0x7f8a8c037c80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f8a9c006f20 tx=0x7f8a9c008040 comp rx=0 tx=0).stop 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.561+0000 7f8aaed12700 10 monclient: shutdown 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.561+0000 7f8aaed12700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.561+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 msgr2=0x7f8aa8136870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.561+0000 7f8aaed12700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 0x7f8aa8136870 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400ea80 tx=0x7f8a940091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 shutdown_connections 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 --2- 172.21.15.6:0/4257108657 >> v2:172.21.15.22:6812/142603 conn(0x7f8a8c0357d0 0x7f8a8c037c80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8aa8137430 0x7f8aa814e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8aa8069580 0x7f8aa814ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 --2- 172.21.15.6:0/4257108657 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8aa809a8f0 0x7f8aa8136870 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 >> 172.21.15.6:0/4257108657 conn(0x7f8aa80a32b0 msgr2=0x7f8aa80a2ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 shutdown_connections 2023-09-30T00:44:33.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.562+0000 7f8aaed12700 1 -- 172.21.15.6:0/4257108657 wait complete. 2023-09-30T00:44:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.565+0000 7fd966fe9700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.565+0000 7fd966fe9700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.565+0000 7fd966fe9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.565+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fd96013adc0 con 0x7fd960069730 2023-09-30T00:44:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.566+0000 7fd95cff9700 1 -- 172.21.15.6:0/1774389629 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd95401c390 con 0x7fd960069730 2023-09-30T00:44:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.566+0000 7fd95cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2023-09-30T00:44:33.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.566+0000 7fd95cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:33.566 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:33.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.568+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 >> v2:172.21.15.22:6812/142603 conn(0x7fd944035820 msgr2=0x7fd944037cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.568+0000 7fd966fe9700 1 --2- 172.21.15.6:0/1774389629 >> v2:172.21.15.22:6812/142603 conn(0x7fd944035820 0x7fd944037cd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd96011c6b0 tx=0x7fd94c007b50 comp rx=0 tx=0).stop 2023-09-30T00:44:33.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.569+0000 7fd966fe9700 10 monclient: shutdown 2023-09-30T00:44:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.569+0000 7fd966fe9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.569+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 msgr2=0x7fd96013ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.569+0000 7fd966fe9700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013ed30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fd95400c5d0 tx=0x7fd954007f40 comp rx=0 tx=0).stop 2023-09-30T00:44:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.570+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 shutdown_connections 2023-09-30T00:44:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.570+0000 7fd966fe9700 1 --2- 172.21.15.6:0/1774389629 >> v2:172.21.15.22:6812/142603 conn(0x7fd944035820 0x7fd944037cd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.570+0000 7fd966fe9700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd96009cd40 0x7fd96011acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.570+0000 7fd966fe9700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd960069730 0x7fd96013ed30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.570+0000 7fd966fe9700 1 --2- 172.21.15.6:0/1774389629 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd96009a8f0 0x7fd96011a770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.570+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 >> 172.21.15.6:0/1774389629 conn(0x7fd9600a32b0 msgr2=0x7fd96006e170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 shutdown_connections 2023-09-30T00:44:33.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7fd966fe9700 1 -- 172.21.15.6:0/1774389629 wait complete. 2023-09-30T00:44:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f847dace700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f847dace700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f847dace700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f847809cfe0 con 0x7f847809dc90 2023-09-30T00:44:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f845f7fe700 1 -- 172.21.15.6:0/2813349789 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f84680108a0 con 0x7f847809dc90 2023-09-30T00:44:33.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f845f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2023-09-30T00:44:33.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.571+0000 7f845f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:33.572 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:33.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 >> v2:172.21.15.22:6812/142603 conn(0x7f84640357f0 msgr2=0x7f8464037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 --2- 172.21.15.6:0/2813349789 >> v2:172.21.15.22:6812/142603 conn(0x7f84640357f0 0x7f8464037ca0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f847809a7c0 tx=0x7f846c016000 comp rx=0 tx=0).stop 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 10 monclient: shutdown 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 msgr2=0x7f8478141c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 0x7f8478141c20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f846800ec80 tx=0x7f84680091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 shutdown_connections 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 --2- 172.21.15.6:0/2813349789 >> v2:172.21.15.22:6812/142603 conn(0x7f84640357f0 0x7f8464037ca0 secure :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f847809a7c0 tx=0x7f846c016000 comp rx=0 tx=0).stop 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f847809e5d0 0x7f8478145ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f847809d250 0x7f8478142590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 --2- 172.21.15.6:0/2813349789 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f847809dc90 0x7f8478141c20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.575+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 >> 172.21.15.6:0/2813349789 conn(0x7f84780a3320 msgr2=0x7f84780a20b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.579+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 shutdown_connections 2023-09-30T00:44:33.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.579+0000 7f847dace700 1 -- 172.21.15.6:0/2813349789 wait complete. 2023-09-30T00:44:33.583 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.5 2023-09-30T00:44:33.583 DEBUG:teuthology.parallel:result is None 2023-09-30T00:44:33.587 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.4 2023-09-30T00:44:33.587 DEBUG:teuthology.parallel:result is None 2023-09-30T00:44:33.593 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.3 2023-09-30T00:44:33.594 DEBUG:teuthology.parallel:result is None 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcc9907700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcc9907700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcc9907700 10 monclient: _send_mon_message to mon.b at v2:172.21.15.22:3300/0 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7fbcc4133930 con 0x7fbcc4098670 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcaaffd700 1 -- 172.21.15.6:0/843218204 <== mon.1 v2:172.21.15.22:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbcc4133930 con 0x7fbcc4098670 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcaaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.598+0000 7fbcaaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:33.598 INFO:teuthology.orchestra.run.smithi006.stdout:47244640260 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 >> v2:172.21.15.22:6812/142603 conn(0x7fbcb00357f0 msgr2=0x7fbcb0037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 --2- 172.21.15.6:0/843218204 >> v2:172.21.15.22:6812/142603 conn(0x7fbcb00357f0 0x7fbcb0037ca0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbcb4009ca0 tx=0x7fbcb4009250 comp rx=0 tx=0).stop 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 10 monclient: shutdown 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 msgr2=0x7fbcc4137400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 0x7fbcc4137400 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fbcbc01dce0 tx=0x7fbcbc005450 comp rx=0 tx=0).stop 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 shutdown_connections 2023-09-30T00:44:33.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 --2- 172.21.15.6:0/843218204 >> v2:172.21.15.22:6812/142603 conn(0x7fbcb00357f0 0x7fbcb0037ca0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbcc4137b60 0x7fbcc409cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbcc4098670 0x7fbcc4137400 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 --2- 172.21.15.6:0/843218204 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbcc4132d00 0x7fbcc409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.600+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 >> 172.21.15.6:0/843218204 conn(0x7fbcc40a5480 msgr2=0x7fbcc40a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.601+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 shutdown_connections 2023-09-30T00:44:33.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.601+0000 7fbcc9907700 1 -- 172.21.15.6:0/843218204 wait complete. 2023-09-30T00:44:33.614 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640260 got 47244640260 for osd.2 2023-09-30T00:44:33.614 DEBUG:teuthology.parallel:result is None 2023-09-30T00:44:33.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd29608d700 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:33.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd29608d700 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:33.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd29608d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:33.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fd270005360 con 0x7fd290098e90 2023-09-30T00:44:33.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd2777fe700 1 -- 172.21.15.6:0/810488304 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fd28801c310 con 0x7fd290098e90 2023-09-30T00:44:33.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd2777fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2023-09-30T00:44:33.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.663+0000 7fd2777fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:33.664 INFO:teuthology.orchestra.run.smithi006.stdout:47244640259 2023-09-30T00:44:33.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 >> v2:172.21.15.22:6812/142603 conn(0x7fd27c035820 msgr2=0x7fd27c037cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 --2- 172.21.15.6:0/810488304 >> v2:172.21.15.22:6812/142603 conn(0x7fd27c035820 0x7fd27c037cd0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fd2840093f0 tx=0x7fd284017000 comp rx=0 tx=0).stop 2023-09-30T00:44:33.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 10 monclient: shutdown 2023-09-30T00:44:33.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 msgr2=0x7fd29012ec80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012ec80 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd28800cfb0 tx=0x7fd2880078c0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 shutdown_connections 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 --2- 172.21.15.6:0/810488304 >> v2:172.21.15.22:6812/142603 conn(0x7fd27c035820 0x7fd27c037cd0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd290098e90 0x7fd29012ec80 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd290098670 0x7fd29009eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 --2- 172.21.15.6:0/810488304 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd290098a40 0x7fd29012e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 >> 172.21.15.6:0/810488304 conn(0x7fd2900a5480 msgr2=0x7fd290133af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 shutdown_connections 2023-09-30T00:44:33.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.666+0000 7fd29608d700 1 -- 172.21.15.6:0/810488304 wait complete. 2023-09-30T00:44:33.679 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640259 got 47244640259 for osd.1 2023-09-30T00:44:33.679 DEBUG:teuthology.parallel:result is None 2023-09-30T00:44:33.679 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:44:33.680 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.811+0000 7fa337a8e700 10 monclient: get_monmap_and_config 2023-09-30T00:44:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.811+0000 7fa337a8e700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.811+0000 7fa337a8e700 10 monclient: monmap: 2023-09-30T00:44:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.811 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:33.811365+0000 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:33.811365+0000 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.811+0000 7fa337a8e700 1 Processor -- start 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 1 -- start start 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient: init 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient: _add_conns ranks=[0,4,3,2,1] 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa330098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa330132d00 0x7fa330137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): picked mon.noname-c con 0x7fa330132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa330137b60 0x7fa330139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): picked mon.noname-a con 0x7fa330137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:33.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3300a1000 con 0x7fa330098670 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa330099150 con 0x7fa330137b60 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3300a0a50 con 0x7fa330132d00 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.812+0000 7fa337a8e700 10 monclient(hunting): authenticate will time out at 1270.109970s 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33582a700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa330098670 0x7fa330098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60952/0 (socket says 172.21.15.6:60952) 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33582a700 1 -- 172.21.15.6:0/3624798187 learned_addr learned my addr 172.21.15.6:0/3624798187 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa330137b60 0x7fa330139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): get_auth_request con 0x7fa330137b60 auth_method 0 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 1 --2- 172.21.15.6:0/3624798187 >> v2:172.21.15.22:3300/0 conn(0x7fa330132d00 0x7fa330137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): get_auth_request con 0x7fa330132d00 auth_method 0 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa335029700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient(hunting): handle_auth_done global_id 4326 payload 274 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 1 -- 172.21.15.6:0/3624798187 >> v2:172.21.15.22:3300/0 conn(0x7fa330132d00 msgr2=0x7fa330137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 1 --2- 172.21.15.6:0/3624798187 >> v2:172.21.15.22:3300/0 conn(0x7fa330132d00 0x7fa330137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 1 -- 172.21.15.6:0/3624798187 >> v1:172.21.15.6:6789/0 conn(0x7fa330098670 legacy=0x7fa330098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 1 monclient: found mon.noname-a 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.813+0000 7fa33602b700 1 -- 172.21.15.6:0/3624798187 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3300a0730 con 0x7fa330137b60 2023-09-30T00:44:33.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.815+0000 7fa33602b700 1 -- 172.21.15.6:0/3624798187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330137b60 msgr2=0x7fa330139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.815+0000 7fa33602b700 1 --2- 172.21.15.6:0/3624798187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330137b60 0x7fa330139f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa330063790 tx=0x7fa328009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81ebd8bd3e17ae71 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.815+0000 7fa334828700 1 -- 172.21.15.6:0/3624798187 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32800e070 con 0x7fa330137b60 2023-09-30T00:44:33.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: dump: 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.825 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 1 -- 172.21.15.6:0/3624798187 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32800bd30 con 0x7fa330137b60 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 5 monclient: authenticate success, global_id 4326 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 1 -- 172.21.15.6:0/3624798187 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32800bea0 con 0x7fa330137b60 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 4 set_mon_vals no callback set 2023-09-30T00:44:33.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa334828700 10 monclient: dump: 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 10 monclient: shutdown 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 -- 172.21.15.6:0/3624798187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330137b60 msgr2=0x7fa330139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 --2- 172.21.15.6:0/3624798187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330137b60 0x7fa330139f40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fa330063790 tx=0x7fa328009780 comp rx=0 tx=0).stop 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 -- 172.21.15.6:0/3624798187 shutdown_connections 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 --2- 172.21.15.6:0/3624798187 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330137b60 0x7fa330139f40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 --2- 172.21.15.6:0/3624798187 >> v2:172.21.15.22:3300/0 conn(0x7fa330132d00 0x7fa330137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 -- 172.21.15.6:0/3624798187 >> 172.21.15.6:0/3624798187 conn(0x7fa3300a5480 msgr2=0x7fa3300a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 -- 172.21.15.6:0/3624798187 shutdown_connections 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.816+0000 7fa337a8e700 1 -- 172.21.15.6:0/3624798187 wait complete. 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.817+0000 7fa337a8e700 10 monclient: build_initial_monmap 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.817+0000 7fa337a8e700 10 monclient: monmap: 2023-09-30T00:44:33.828 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.817+0000 7fa337a8e700 1 Processor -- start 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.817+0000 7fa337a8e700 1 -- start start 2023-09-30T00:44:33.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.817+0000 7fa337a8e700 10 monclient: init 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 0x7fa33012f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): picked mon.noname-a con 0x7fa330098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa330132d00 0x7fa33009ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): picked mon.noname-b con 0x7fa330132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa330137b60 0x7fa3301369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): picked mon.noname-c con 0x7fa330137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3300a1500 con 0x7fa330098670 2023-09-30T00:44:33.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa330137570 con 0x7fa330132d00 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa33009dc80 con 0x7fa330137b60 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa337a8e700 10 monclient(hunting): authenticate will time out at 1270.115138s 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 0x7fa33012f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 0x7fa33012f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42262/0 (socket says 172.21.15.6:42262) 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 -- 172.21.15.6:0/1589897608 learned_addr learned my addr 172.21.15.6:0/1589897608 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): get_auth_request con 0x7fa330098670 auth_method 0 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa330137b60 0x7fa3301369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): get_auth_request con 0x7fa330137b60 auth_method 0 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa330132d00 0x7fa33009ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): get_auth_request con 0x7fa330132d00 auth_method 0 2023-09-30T00:44:33.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33602b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa335029700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient(hunting): handle_auth_done global_id 4329 payload 824 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 -- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa330137b60 msgr2=0x7fa3301369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa330137b60 0x7fa3301369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 -- 172.21.15.6:0/1589897608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa330132d00 msgr2=0x7fa33009ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa330132d00 0x7fa33009ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient: _finish_hunting 0 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 monclient: found mon.noname-a 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.818+0000 7fa33582a700 1 -- 172.21.15.6:0/1589897608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa328009a20 con 0x7fa330098670 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.820+0000 7fa33582a700 1 -- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 msgr2=0x7fa33012f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.820+0000 7fa33582a700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 0x7fa33012f110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00c590 tx=0x7fa32c004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c00cea0 con 0x7fa330098670 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: dump: 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa32c005d40 con 0x7fa330098670 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 5 monclient: authenticate success, global_id 4329 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 10 monclient: _renew_subs 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa33009f230 con 0x7fa330098670 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa32c007300 con 0x7fa330098670 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa325ffb700 10 monclient: dump: 2023-09-30T00:44:33.836 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 10 monclient: _renew_subs 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa327fff700 4 set_mon_vals no callback set 2023-09-30T00:44:33.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.821+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa33009e460 con 0x7fa330098670 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa32c01b3b0 con 0x7fa330098670 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa337a8e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa337a8e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa337a8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa330072b00 con 0x7fa330098670 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa325ffb700 1 --2- 172.21.15.6:0/1589897608 >> v2:172.21.15.22:6812/142603 conn(0x7fa3140357f0 0x7fa314037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa32c046bd0 con 0x7fa330098670 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa335029700 1 --2- 172.21.15.6:0/1589897608 >> v2:172.21.15.22:6812/142603 conn(0x7fa3140357f0 0x7fa314037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:33.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.822+0000 7fa335029700 10 monclient: get_auth_request con 0x7fa3140357f0 auth_method 0 2023-09-30T00:44:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.825+0000 7fa335029700 1 --2- 172.21.15.6:0/1589897608 >> v2:172.21.15.22:6812/142603 conn(0x7fa3140357f0 0x7fa314037ca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa33009f9a0 tx=0x7fa3200091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.827+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa32c01b660 con 0x7fa330098670 2023-09-30T00:44:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.827+0000 7fa325ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:33.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:33.827+0000 7fa325ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:34.028 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.028+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa33009e460 con 0x7fa3140357f0 2023-09-30T00:44:34.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.030+0000 7fa325ffb700 1 -- 172.21.15.6:0/1589897608 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40483 (secure 0 0 0) 0x7fa33009e460 con 0x7fa3140357f0 2023-09-30T00:44:34.031 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 >> v2:172.21.15.22:6812/142603 conn(0x7fa3140357f0 msgr2=0x7fa314037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 --2- 172.21.15.6:0/1589897608 >> v2:172.21.15.22:6812/142603 conn(0x7fa3140357f0 0x7fa314037ca0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa33009f9a0 tx=0x7fa3200091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 10 monclient: shutdown 2023-09-30T00:44:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 msgr2=0x7fa33012f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.033 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 0x7fa33012f110 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa32c00c590 tx=0x7fa32c004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 shutdown_connections 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 --2- 172.21.15.6:0/1589897608 >> v2:172.21.15.22:6812/142603 conn(0x7fa3140357f0 0x7fa314037ca0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa330137b60 0x7fa3301369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa330098670 0x7fa33012f110 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 --2- 172.21.15.6:0/1589897608 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa330132d00 0x7fa33009ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 >> 172.21.15.6:0/1589897608 conn(0x7fa3300a5480 msgr2=0x7fa33012fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.032+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 shutdown_connections 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.033+0000 7fa337a8e700 1 -- 172.21.15.6:0/1589897608 wait complete. 2023-09-30T00:44:34.034 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:44:34.045 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-09-30T00:44:33.537284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31792,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":30784,"kb_avail":566199248,"statfs":{"total":579820584960,"available":579788029952,"internally_reserved":0,"allocated":983040,"data_stored":1314998,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31522816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.117443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.427091+0000","last_change":"2023-09-30T00:44:28.427091+0000","last_active":"2023-09-30T00:44:28.427091+0000","last_peered":"2023-09-30T00:44:28.427091+0000","last_clean":"2023-09-30T00:44:28.427091+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:44:28.427091+0000","last_undegraded":"2023-09-30T00:44:28.427091+0000","last_fullsized":"2023-09-30T00:44:28.427091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:47:25.053141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426953+0000","last_change":"2023-09-30T00:44:28.426953+0000","last_active":"2023-09-30T00:44:28.426953+0000","last_peered":"2023-09-30T00:44:28.426953+0000","last_clean":"2023-09-30T00:44:28.426953+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:44:28.426953+0000","last_undegraded":"2023-09-30T00:44:28.426953+0000","last_fullsized":"2023-09-30T00:44:28.426953+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:21:01.653238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423208+0000","last_change":"2023-09-30T00:44:28.423208+0000","last_active":"2023-09-30T00:44:28.423208+0000","last_peered":"2023-09-30T00:44:28.423208+0000","last_clean":"2023-09-30T00:44:28.423208+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:44:28.423208+0000","last_undegraded":"2023-09-30T00:44:28.423208+0000","last_fullsized":"2023-09-30T00:44:28.423208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:04:55.556087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426680+0000","last_change":"2023-09-30T00:44:28.423004+0000","last_active":"2023-09-30T00:44:28.426680+0000","last_peered":"2023-09-30T00:44:28.426680+0000","last_clean":"2023-09-30T00:44:28.426680+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:44:28.426680+0000","last_undegraded":"2023-09-30T00:44:28.426680+0000","last_fullsized":"2023-09-30T00:44:28.426680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:22:00.541043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":161,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:44:27.452452+0000","last_change":"2023-09-30T00:44:25.575831+0000","last_active":"2023-09-30T00:44:27.452452+0000","last_peered":"2023-09-30T00:44:27.452452+0000","last_clean":"2023-09-30T00:44:27.452452+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:44:27.452452+0000","last_undegraded":"2023-09-30T00:44:27.452452+0000","last_fullsized":"2023-09-30T00:44:27.452452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_clean_scrub_stamp":"2023-09-30T00:44:24.231195+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:05.898656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.422311+0000","last_change":"2023-09-30T00:44:28.422311+0000","last_active":"2023-09-30T00:44:28.422311+0000","last_peered":"2023-09-30T00:44:28.422311+0000","last_clean":"2023-09-30T00:44:28.422311+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:44:28.422311+0000","last_undegraded":"2023-09-30T00:44:28.422311+0000","last_fullsized":"2023-09-30T00:44:28.422311+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:37:58.120817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423390+0000","last_change":"2023-09-30T00:44:28.422342+0000","last_active":"2023-09-30T00:44:28.423390+0000","last_peered":"2023-09-30T00:44:28.423390+0000","last_clean":"2023-09-30T00:44:28.423390+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:44:28.423390+0000","last_undegraded":"2023-09-30T00:44:28.423390+0000","last_fullsized":"2023-09-30T00:44:28.423390+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:26.637840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426632+0000","last_change":"2023-09-30T00:44:28.424009+0000","last_active":"2023-09-30T00:44:28.426632+0000","last_peered":"2023-09-30T00:44:28.426632+0000","last_clean":"2023-09-30T00:44:28.426632+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:44:28.426632+0000","last_undegraded":"2023-09-30T00:44:28.426632+0000","last_fullsized":"2023-09-30T00:44:28.426632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:57.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424000+0000","last_change":"2023-09-30T00:44:28.424000+0000","last_active":"2023-09-30T00:44:28.424000+0000","last_peered":"2023-09-30T00:44:28.424000+0000","last_clean":"2023-09-30T00:44:28.424000+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:44:28.424000+0000","last_undegraded":"2023-09-30T00:44:28.424000+0000","last_fullsized":"2023-09-30T00:44:28.424000+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:57:56.916213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":569769,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":40841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":569769,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:34.047 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.178+0000 7f0ac534b700 10 monclient: get_monmap_and_config 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.178+0000 7f0ac534b700 10 monclient: build_initial_monmap 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.178+0000 7f0ac534b700 10 monclient: monmap: 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:34.178486+0000 2023-09-30T00:44:34.178 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:34.178486+0000 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:34.179 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.178+0000 7f0ac534b700 1 Processor -- start 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.178+0000 7f0ac534b700 1 -- start start 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.178+0000 7f0ac534b700 10 monclient: init 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0ac0132150 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 0x7f0ac0132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): picked mon.noname-a con 0x7f0ac0132e10 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 0x7f0ac01394f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): picked mon.noname-b con 0x7f0ac0098e70 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ac0132520 con 0x7f0ac0132150 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ac0096a80 con 0x7f0ac0132e10 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac00a0f80 con 0x7f0ac0098e70 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0ac534b700 10 monclient(hunting): authenticate will time out at 1270.476501s 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 0x7f0ac01394f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 0x7f0ac0132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 0x7f0ac01394f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36790/0 (socket says 172.21.15.6:36790) 2023-09-30T00:44:34.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 1 -- 172.21.15.6:0/2220306890 learned_addr learned my addr 172.21.15.6:0/2220306890 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 0x7f0ac0132790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42270/0 (socket says 172.21.15.6:42270) 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): get_auth_request con 0x7f0ac0098e70 auth_method 0 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): get_auth_request con 0x7f0ac0132e10 auth_method 0 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 10 monclient(hunting): handle_auth_done global_id 4325 payload 274 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 1 -- 172.21.15.6:0/2220306890 >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 msgr2=0x7f0ac0132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 1 --2- 172.21.15.6:0/2220306890 >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 0x7f0ac0132790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.179+0000 7f0abf7fe700 1 -- 172.21.15.6:0/2220306890 >> v1:172.21.15.22:6789/0 conn(0x7f0ac0132150 legacy=0x7f0ac0098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.180+0000 7f0abf7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.180+0000 7f0abf7fe700 1 monclient: found mon.noname-b 2023-09-30T00:44:34.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.180+0000 7f0abf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.180+0000 7f0abf7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.180+0000 7f0abf7fe700 1 -- 172.21.15.6:0/2220306890 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ac0096d20 con 0x7f0ac0098e70 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.180+0000 7f0abe7fc700 1 --2- 172.21.15.6:0/2220306890 >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 0x7f0ac0132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abf7fe700 1 -- 172.21.15.6:0/2220306890 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 msgr2=0x7f0ac01394f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abf7fe700 1 --2- 172.21.15.6:0/2220306890 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 0x7f0ac01394f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009f20 tx=0x7f0ab8009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c4ac22106f33a47f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 1 -- 172.21.15.6:0/2220306890 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800e070 con 0x7f0ac0098e70 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: dump: 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.187 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: _finish_auth 0 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 5 monclient: authenticate success, global_id 4325 2023-09-30T00:44:34.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 1 -- 172.21.15.6:0/2220306890 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab800bba0 con 0x7f0ac0098e70 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 1 -- 172.21.15.6:0/2220306890 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab80092e0 con 0x7f0ac0098e70 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 4 set_mon_vals no callback set 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0abdffb700 10 monclient: dump: 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.189 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 10 monclient: shutdown 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 1 -- 172.21.15.6:0/2220306890 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 msgr2=0x7f0ac01394f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.182+0000 7f0ac534b700 1 --2- 172.21.15.6:0/2220306890 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 0x7f0ac01394f0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0ab8009f20 tx=0x7f0ab8009840 comp rx=0 tx=0).stop 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 1 -- 172.21.15.6:0/2220306890 shutdown_connections 2023-09-30T00:44:34.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 1 --2- 172.21.15.6:0/2220306890 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0098e70 0x7f0ac01394f0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 1 --2- 172.21.15.6:0/2220306890 >> v2:172.21.15.6:3300/0 conn(0x7f0ac0132e10 0x7f0ac0132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 1 -- 172.21.15.6:0/2220306890 >> 172.21.15.6:0/2220306890 conn(0x7f0ac00a54c0 msgr2=0x7f0ac00a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 1 -- 172.21.15.6:0/2220306890 shutdown_connections 2023-09-30T00:44:34.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 1 -- 172.21.15.6:0/2220306890 wait complete. 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 10 monclient: build_initial_monmap 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.183+0000 7f0ac534b700 10 monclient: monmap: 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:34.195 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:34.196 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 Processor -- start 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 -- start start 2023-09-30T00:44:34.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient: init 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ac0098e70 0x7f0ac009bc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): picked mon.noname-a con 0x7f0ac0098e70 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 0x7f0ac01332b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): picked mon.noname-c con 0x7f0ac0132150 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ac0132e10 0x7f0ac01337f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): picked mon.noname-b con 0x7f0ac0132e10 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01360b0 con 0x7f0ac0098e70 2023-09-30T00:44:34.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac0134b70 con 0x7f0ac0132e10 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ac01396e0 con 0x7f0ac0132150 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0ac534b700 10 monclient(hunting): authenticate will time out at 1270.481683s 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 0x7f0ac01332b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ac0098e70 0x7f0ac009bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 0x7f0ac01332b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36800/0 (socket says 172.21.15.6:36800) 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): get_auth_request con 0x7f0ac0098e70 auth_method 0 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 1 -- 172.21.15.6:0/3701158513 learned_addr learned my addr 172.21.15.6:0/3701158513 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:34.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): get_auth_request con 0x7f0ac0132150 auth_method 0 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.184+0000 7f0abeffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abf7fe700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ac0132e10 0x7f0ac01337f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abf7fe700 10 monclient(hunting): get_auth_request con 0x7f0ac0132e10 auth_method 0 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 10 monclient(hunting): handle_auth_done global_id 4328 payload 824 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 1 -- 172.21.15.6:0/3701158513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ac0132e10 msgr2=0x7f0ac01337f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ac0132e10 0x7f0ac01337f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 1 -- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ac0098e70 msgr2=0x7f0ac009bc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ac0098e70 0x7f0ac009bc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 1 monclient: found mon.noname-c 2023-09-30T00:44:34.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abe7fc700 1 -- 172.21.15.6:0/3701158513 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab4009420 con 0x7f0ac0132150 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abeffd700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ac0098e70 0x7f0ac009bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.185+0000 7f0abf7fe700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ac0132e10 0x7f0ac01337f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0abe7fc700 1 -- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 msgr2=0x7f0ac01332b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0abe7fc700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 0x7f0ac01332b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400ec80 tx=0x7f0ab40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab400cb30 con 0x7f0ac0132150 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: dump: 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.202 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 5 monclient: authenticate success, global_id 4328 2023-09-30T00:44:34.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0ab400cc90 con 0x7f0ac0132150 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ab40136c0 con 0x7f0ac0132150 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 10 monclient: _renew_subs 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab8009ae0 con 0x7f0ac0132150 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0abd7fa700 4 set_mon_vals no callback set 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0aa6ffd700 10 monclient: dump: 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.204 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 10 monclient: _renew_subs 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.187+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ac0134580 con 0x7f0ac0132150 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0ab4013a30 con 0x7f0ac0132150 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0ac534b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:34.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0ac534b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0ac534b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ac0072b00 con 0x7f0ac0132150 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0aa6ffd700 1 --2- 172.21.15.6:0/3701158513 >> v2:172.21.15.22:6812/142603 conn(0x7f0aac0357d0 0x7f0aac037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.188+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0ab4047a50 con 0x7f0ac0132150 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.189+0000 7f0abeffd700 1 --2- 172.21.15.6:0/3701158513 >> v2:172.21.15.22:6812/142603 conn(0x7f0aac0357d0 0x7f0aac037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.189+0000 7f0abeffd700 10 monclient: get_auth_request con 0x7f0aac0357d0 auth_method 0 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.191+0000 7f0abeffd700 1 --2- 172.21.15.6:0/3701158513 >> v2:172.21.15.22:6812/142603 conn(0x7f0aac0357d0 0x7f0aac037c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0002e00 tx=0x7f0ab0007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.193+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0ab4013ce0 con 0x7f0ac0132150 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.193+0000 7f0aa6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:34.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.193+0000 7f0aa6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:34.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.390+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0ac0063990 con 0x7f0aac0357d0 2023-09-30T00:44:34.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.392+0000 7f0aa6ffd700 1 -- 172.21.15.6:0/3701158513 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+40483 (secure 0 0 0) 0x7f0ac0063990 con 0x7f0aac0357d0 2023-09-30T00:44:34.393 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 >> v2:172.21.15.22:6812/142603 conn(0x7f0aac0357d0 msgr2=0x7f0aac037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 --2- 172.21.15.6:0/3701158513 >> v2:172.21.15.22:6812/142603 conn(0x7f0aac0357d0 0x7f0aac037c80 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0ab0002e00 tx=0x7f0ab0007ea0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 10 monclient: shutdown 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 msgr2=0x7f0ac01332b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 0x7f0ac01332b0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f0ab400ec80 tx=0x7f0ab40091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 shutdown_connections 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 --2- 172.21.15.6:0/3701158513 >> v2:172.21.15.22:6812/142603 conn(0x7f0aac0357d0 0x7f0aac037c80 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0ac0132e10 0x7f0ac01337f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0ac0098e70 0x7f0ac009bc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 --2- 172.21.15.6:0/3701158513 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0ac0132150 0x7f0ac01332b0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 >> 172.21.15.6:0/3701158513 conn(0x7f0ac00a54c0 msgr2=0x7f0ac0097460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:34.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.394+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 shutdown_connections 2023-09-30T00:44:34.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.395+0000 7f0ac534b700 1 -- 172.21.15.6:0/3701158513 wait complete. 2023-09-30T00:44:34.396 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:44:34.407 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2023-09-30T00:44:33.537284+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31792,"kb_used_data":960,"kb_used_omap":0,"kb_used_meta":30784,"kb_avail":566199248,"statfs":{"total":579820584960,"available":579788029952,"internally_reserved":0,"allocated":983040,"data_stored":1314998,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31522816},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.117443"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.427091+0000","last_change":"2023-09-30T00:44:28.427091+0000","last_active":"2023-09-30T00:44:28.427091+0000","last_peered":"2023-09-30T00:44:28.427091+0000","last_clean":"2023-09-30T00:44:28.427091+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:44:28.427091+0000","last_undegraded":"2023-09-30T00:44:28.427091+0000","last_fullsized":"2023-09-30T00:44:28.427091+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:47:25.053141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426953+0000","last_change":"2023-09-30T00:44:28.426953+0000","last_active":"2023-09-30T00:44:28.426953+0000","last_peered":"2023-09-30T00:44:28.426953+0000","last_clean":"2023-09-30T00:44:28.426953+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:44:28.426953+0000","last_undegraded":"2023-09-30T00:44:28.426953+0000","last_fullsized":"2023-09-30T00:44:28.426953+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:21:01.653238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423208+0000","last_change":"2023-09-30T00:44:28.423208+0000","last_active":"2023-09-30T00:44:28.423208+0000","last_peered":"2023-09-30T00:44:28.423208+0000","last_clean":"2023-09-30T00:44:28.423208+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:44:28.423208+0000","last_undegraded":"2023-09-30T00:44:28.423208+0000","last_fullsized":"2023-09-30T00:44:28.423208+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:04:55.556087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426680+0000","last_change":"2023-09-30T00:44:28.423004+0000","last_active":"2023-09-30T00:44:28.426680+0000","last_peered":"2023-09-30T00:44:28.426680+0000","last_clean":"2023-09-30T00:44:28.426680+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:44:28.426680+0000","last_undegraded":"2023-09-30T00:44:28.426680+0000","last_fullsized":"2023-09-30T00:44:28.426680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:22:00.541043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":161,"reported_epoch":15,"state":"active+clean","last_fresh":"2023-09-30T00:44:27.452452+0000","last_change":"2023-09-30T00:44:25.575831+0000","last_active":"2023-09-30T00:44:27.452452+0000","last_peered":"2023-09-30T00:44:27.452452+0000","last_clean":"2023-09-30T00:44:27.452452+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:44:27.452452+0000","last_undegraded":"2023-09-30T00:44:27.452452+0000","last_fullsized":"2023-09-30T00:44:27.452452+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_clean_scrub_stamp":"2023-09-30T00:44:24.231195+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:05.898656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.422311+0000","last_change":"2023-09-30T00:44:28.422311+0000","last_active":"2023-09-30T00:44:28.422311+0000","last_peered":"2023-09-30T00:44:28.422311+0000","last_clean":"2023-09-30T00:44:28.422311+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:44:28.422311+0000","last_undegraded":"2023-09-30T00:44:28.422311+0000","last_fullsized":"2023-09-30T00:44:28.422311+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:37:58.120817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423390+0000","last_change":"2023-09-30T00:44:28.422342+0000","last_active":"2023-09-30T00:44:28.423390+0000","last_peered":"2023-09-30T00:44:28.423390+0000","last_clean":"2023-09-30T00:44:28.423390+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:44:28.423390+0000","last_undegraded":"2023-09-30T00:44:28.423390+0000","last_fullsized":"2023-09-30T00:44:28.423390+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:26.637840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426632+0000","last_change":"2023-09-30T00:44:28.424009+0000","last_active":"2023-09-30T00:44:28.426632+0000","last_peered":"2023-09-30T00:44:28.426632+0000","last_clean":"2023-09-30T00:44:28.426632+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:44:28.426632+0000","last_undegraded":"2023-09-30T00:44:28.426632+0000","last_fullsized":"2023-09-30T00:44:28.426632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:57.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":8,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424000+0000","last_change":"2023-09-30T00:44:28.424000+0000","last_active":"2023-09-30T00:44:28.424000+0000","last_peered":"2023-09-30T00:44:28.424000+0000","last_clean":"2023-09-30T00:44:28.424000+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:44:28.424000+0000","last_undegraded":"2023-09-30T00:44:28.424000+0000","last_fullsized":"2023-09-30T00:44:28.424000+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:57:56.916213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":569769,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":4,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5072,"kb_used_data":136,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":94366768,"statfs":{"total":96636764160,"available":96631570432,"internally_reserved":0,"allocated":139264,"data_stored":40841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39400000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60499999999999998}]}]},{"osd":3,"up_from":11,"seq":47244640259,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65800000000000003}]}]},{"osd":2,"up_from":11,"seq":47244640260,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51500000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49299999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83099999999999996}]}]},{"osd":0,"up_from":11,"seq":47244640259,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54000000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56699999999999995}]}]},{"osd":1,"up_from":11,"seq":47244640259,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5644,"kb_used_data":196,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366196,"statfs":{"total":96636764160,"available":96630984704,"internally_reserved":0,"allocated":200704,"data_stored":569769,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:44:34.409 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:44:34.409 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-09-30T00:44:34.409 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2023-09-30T00:44:34.409 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:34.538 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.539+0000 7f0f9af48700 10 monclient: get_monmap_and_config 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.539+0000 7f0f9af48700 10 monclient: build_initial_monmap 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.539+0000 7f0f9af48700 10 monclient: monmap: 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:34.539411+0000 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:34.539411+0000 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.539 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:34.540 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:34.540 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:34.540 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:34.540 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:34.540 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 Processor -- start 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 -- start start 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient: init 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f94132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): picked mon.noname-b con 0x7f0f94098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:34.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0f94098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f0f94098e90 0x7f0f9412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): picked mon.noname-c con 0x7f0f94098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f940a1000 con 0x7f0f94098a40 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f940a0a50 con 0x7f0f94098e90 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f940a0730 con 0x7f0f94098670 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f9af48700 10 monclient(hunting): authenticate will time out at 1270.837690s 2023-09-30T00:44:34.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f94132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f94132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36816/0 (socket says 172.21.15.6:36816) 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 1 -- 172.21.15.6:0/1981334179 learned_addr learned my addr 172.21.15.6:0/1981334179 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): get_auth_request con 0x7f0f94098670 auth_method 0 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.540+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f994e5700 1 --2- 172.21.15.6:0/1981334179 >> v2:172.21.15.22:3300/0 conn(0x7f0f94098e90 0x7f0f9412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f994e5700 10 monclient(hunting): get_auth_request con 0x7f0f94098e90 auth_method 0 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f994e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f994e5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f994e5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_done global_id 4331 payload 274 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 1 -- 172.21.15.6:0/1981334179 >> v2:172.21.15.22:3300/0 conn(0x7f0f94098e90 msgr2=0x7f0f9412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 1 --2- 172.21.15.6:0/1981334179 >> v2:172.21.15.22:3300/0 conn(0x7f0f94098e90 0x7f0f9412f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 1 -- 172.21.15.6:0/1981334179 >> v1:172.21.15.22:6789/0 conn(0x7f0f94098a40 legacy=0x7f0f9412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 10 monclient: _finish_hunting 0 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 1 monclient: found mon.noname-b 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:34.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f98ce4700 1 -- 172.21.15.6:0/1981334179 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f94068df0 con 0x7f0f94098670 2023-09-30T00:44:34.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.541+0000 7f0f994e5700 1 --2- 172.21.15.6:0/1981334179 >> v2:172.21.15.22:3300/0 conn(0x7f0f94098e90 0x7f0f9412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:34.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f98ce4700 1 -- 172.21.15.6:0/1981334179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 msgr2=0x7f0f94132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:34.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f98ce4700 1 --2- 172.21.15.6:0/1981334179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f94132d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0f900099b0 tx=0x7f0f90009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a0e94239e06b1cb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:34.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 1 -- 172.21.15.6:0/1981334179 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9000ba30 con 0x7f0f94098670 2023-09-30T00:44:34.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 10 monclient: dump: 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.553 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f9af48700 5 monclient: authenticate success, global_id 4331 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f9af48700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 1 -- 172.21.15.6:0/1981334179 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9000bb90 con 0x7f0f94098670 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.543+0000 7f0f8b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:34.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 4 set_mon_vals no callback set 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f8b7fe700 1 -- 172.21.15.6:0/1981334179 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f90011420 con 0x7f0f94098670 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f8b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f8b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f8b7fe700 10 monclient: dump: 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.555 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 10 monclient: shutdown 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 -- 172.21.15.6:0/1981334179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 msgr2=0x7f0f94132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 --2- 172.21.15.6:0/1981334179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f94132d00 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f0f900099b0 tx=0x7f0f90009250 comp rx=0 tx=0).stop 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 -- 172.21.15.6:0/1981334179 shutdown_connections 2023-09-30T00:44:34.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 --2- 172.21.15.6:0/1981334179 >> v2:172.21.15.22:3300/0 conn(0x7f0f94098e90 0x7f0f9412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 --2- 172.21.15.6:0/1981334179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f94132d00 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 -- 172.21.15.6:0/1981334179 >> 172.21.15.6:0/1981334179 conn(0x7f0f940a5480 msgr2=0x7f0f940a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 -- 172.21.15.6:0/1981334179 shutdown_connections 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.544+0000 7f0f9af48700 1 -- 172.21.15.6:0/1981334179 wait complete. 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.545+0000 7f0f9af48700 10 monclient: build_initial_monmap 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.545+0000 7f0f9af48700 10 monclient: monmap: 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:34.557 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.545+0000 7f0f9af48700 1 Processor -- start 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.545+0000 7f0f9af48700 1 -- start start 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.545+0000 7f0f9af48700 10 monclient: init 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f941417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): picked mon.noname-c con 0x7f0f94098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0f94098a40 0x7f0f9412c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): picked mon.noname-b con 0x7f0f94098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94098e90 0x7f0f9409cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): picked mon.noname-a con 0x7f0f94098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9411c230 con 0x7f0f94098e90 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f94131300 con 0x7f0f94098a40 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f9409f1c0 con 0x7f0f94098670 2023-09-30T00:44:34.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f9af48700 10 monclient(hunting): authenticate will time out at 1270.843267s 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f941417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94098e90 0x7f0f9409cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f941417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36820/0 (socket says 172.21.15.6:36820) 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): get_auth_request con 0x7f0f94098e90 auth_method 0 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 -- 172.21.15.6:0/2028577110 learned_addr learned my addr 172.21.15.6:0/2028577110 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): get_auth_request con 0x7f0f94098670 auth_method 0 2023-09-30T00:44:34.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f8bfff700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0f94098a40 0x7f0f9412c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f8bfff700 10 monclient(hunting): get_auth_request con 0x7f0f94098a40 auth_method 0 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f8bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:34.561 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f8bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f8bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient(hunting): handle_auth_done global_id 4334 payload 824 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 -- 172.21.15.6:0/2028577110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0f94098a40 msgr2=0x7f0f9412c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0f94098a40 0x7f0f9412c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 -- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94098e90 msgr2=0x7f0f9409cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94098e90 0x7f0f9409cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient: _finish_hunting 0 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 monclient: found mon.noname-c 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.562 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f98ce4700 1 -- 172.21.15.6:0/2028577110 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f900094f0 con 0x7f0f94098670 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f994e5700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94098e90 0x7f0f9409cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.546+0000 7f0f8bfff700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0f94098a40 0x7f0f9412c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f98ce4700 1 -- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 msgr2=0x7f0f941417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f98ce4700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f941417d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f900090b0 tx=0x7f0f90016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9001a430 con 0x7f0f94098670 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: dump: 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.563 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f9001a5b0 con 0x7f0f94098670 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 5 monclient: authenticate success, global_id 4334 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 10 monclient: _renew_subs 2023-09-30T00:44:34.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f940a1500 con 0x7f0f94098670 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0f9001a970 con 0x7f0f94098670 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f88ff9700 10 monclient: dump: 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.565 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 10 monclient: _renew_subs 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f8affd700 4 set_mon_vals no callback set 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.549+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f9409e580 con 0x7f0f94098670 2023-09-30T00:44:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f9af48700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f9af48700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f9af48700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f9411c730 con 0x7f0f94098670 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0f900253f0 con 0x7f0f94098670 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f88ff9700 1 --2- 172.21.15.6:0/2028577110 >> v2:172.21.15.22:6812/142603 conn(0x7f0f800357a0 0x7f0f80037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0f90049500 con 0x7f0f94098670 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f8bfff700 1 --2- 172.21.15.6:0/2028577110 >> v2:172.21.15.22:6812/142603 conn(0x7f0f800357a0 0x7f0f80037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.550+0000 7f0f8bfff700 10 monclient: get_auth_request con 0x7f0f800357a0 auth_method 0 2023-09-30T00:44:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.553+0000 7f0f8bfff700 1 --2- 172.21.15.6:0/2028577110 >> v2:172.21.15.22:6812/142603 conn(0x7f0f800357a0 0x7f0f80037c50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f84007fd0 tx=0x7f0f84009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:34.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.555+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0f90018350 con 0x7f0f94098670 2023-09-30T00:44:34.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.555+0000 7f0f88ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:34.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.555+0000 7f0f88ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f9af48700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f9af48700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f9af48700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0f94072b00 con 0x7f0f94098670 2023-09-30T00:44:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f88ff9700 1 -- 172.21.15.6:0/2028577110 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0f9002db90 con 0x7f0f94098670 2023-09-30T00:44:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f88ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.799+0000 7f0f88ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:34.800 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:34.800 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 >> v2:172.21.15.22:6812/142603 conn(0x7f0f800357a0 msgr2=0x7f0f80037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 --2- 172.21.15.6:0/2028577110 >> v2:172.21.15.22:6812/142603 conn(0x7f0f800357a0 0x7f0f80037c50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0f84007fd0 tx=0x7f0f84009040 comp rx=0 tx=0).stop 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 10 monclient: shutdown 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 msgr2=0x7f0f941417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f941417d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f0f900090b0 tx=0x7f0f90016000 comp rx=0 tx=0).stop 2023-09-30T00:44:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 shutdown_connections 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 --2- 172.21.15.6:0/2028577110 >> v2:172.21.15.22:6812/142603 conn(0x7f0f800357a0 0x7f0f80037c50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0f94098e90 0x7f0f9409cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0f94098670 0x7f0f941417d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 --2- 172.21.15.6:0/2028577110 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0f94098a40 0x7f0f9412c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 >> 172.21.15.6:0/2028577110 conn(0x7f0f940a5480 msgr2=0x7f0f941306f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 shutdown_connections 2023-09-30T00:44:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:34.802+0000 7f0f9af48700 1 -- 172.21.15.6:0/2028577110 wait complete. 2023-09-30T00:44:37.816 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.954+0000 7fce36de2700 10 monclient: get_monmap_and_config 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.954+0000 7fce36de2700 10 monclient: build_initial_monmap 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.954+0000 7fce36de2700 10 monclient: monmap: 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:37.954482+0000 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:37.954482+0000 2023-09-30T00:44:37.954 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:37.955 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:37.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 Processor -- start 2023-09-30T00:44:37.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 -- start start 2023-09-30T00:44:37.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient: init 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient: _add_conns ranks=[0,4,3,1,2] 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fce30098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fce30132d00 0x7fce30137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): picked mon.noname-c con 0x7fce30132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce30137b60 0x7fce30139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): picked mon.noname-a con 0x7fce30137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce300a1000 con 0x7fce30098670 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce30099150 con 0x7fce30137b60 2023-09-30T00:44:37.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce300a0a50 con 0x7fce30132d00 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce36de2700 10 monclient(hunting): authenticate will time out at 1274.252789s 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce3537f700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce30137b60 0x7fce30139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce34b7e700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fce30098670 0x7fce30098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50296/0 (socket says 172.21.15.6:50296) 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce34b7e700 1 -- 172.21.15.6:0/1159651257 learned_addr learned my addr 172.21.15.6:0/1159651257 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce3537f700 10 monclient(hunting): get_auth_request con 0x7fce30137b60 auth_method 0 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce3537f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce3537f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.955+0000 7fce3537f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:37.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 1 --2- 172.21.15.6:0/1159651257 >> v2:172.21.15.22:3300/0 conn(0x7fce30132d00 0x7fce30137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): get_auth_request con 0x7fce30132d00 auth_method 0 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce2ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 10 monclient(hunting): handle_auth_done global_id 4341 payload 274 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 1 -- 172.21.15.6:0/1159651257 >> v2:172.21.15.22:3300/0 conn(0x7fce30132d00 msgr2=0x7fce30137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:37.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 1 --2- 172.21.15.6:0/1159651257 >> v2:172.21.15.22:3300/0 conn(0x7fce30132d00 0x7fce30137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 1 -- 172.21.15.6:0/1159651257 >> v1:172.21.15.6:6789/0 conn(0x7fce30098670 legacy=0x7fce30098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 10 monclient: _finish_hunting 0 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 1 monclient: found mon.noname-a 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.956+0000 7fce3537f700 1 -- 172.21.15.6:0/1159651257 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce300a0730 con 0x7fce30137b60 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce3537f700 1 -- 172.21.15.6:0/1159651257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30137b60 msgr2=0x7fce30139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce3537f700 1 --2- 172.21.15.6:0/1159651257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30137b60 0x7fce30139f40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce30063790 tx=0x7fce28009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2da313e1a485eb1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 1 -- 172.21.15.6:0/1159651257 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2800e070 con 0x7fce30137b60 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:37.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: dump: 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:37.964 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 1 -- 172.21.15.6:0/1159651257 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2800bd30 con 0x7fce30137b60 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 1 -- 172.21.15.6:0/1159651257 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2800bea0 con 0x7fce30137b60 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce36de2700 5 monclient: authenticate success, global_id 4341 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce36de2700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce36de2700 4 set_mon_vals no callback set 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.958+0000 7fce2f7fe700 10 monclient: dump: 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:37.965 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 10 monclient: shutdown 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 -- 172.21.15.6:0/1159651257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30137b60 msgr2=0x7fce30139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:37.966 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 --2- 172.21.15.6:0/1159651257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30137b60 0x7fce30139f40 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fce30063790 tx=0x7fce28009780 comp rx=0 tx=0).stop 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 -- 172.21.15.6:0/1159651257 shutdown_connections 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 --2- 172.21.15.6:0/1159651257 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30137b60 0x7fce30139f40 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 --2- 172.21.15.6:0/1159651257 >> v2:172.21.15.22:3300/0 conn(0x7fce30132d00 0x7fce30137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 -- 172.21.15.6:0/1159651257 >> 172.21.15.6:0/1159651257 conn(0x7fce300a5480 msgr2=0x7fce300a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 -- 172.21.15.6:0/1159651257 shutdown_connections 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 1 -- 172.21.15.6:0/1159651257 wait complete. 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 10 monclient: build_initial_monmap 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.959+0000 7fce36de2700 10 monclient: monmap: 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:37.967 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:37.968 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 1 Processor -- start 2023-09-30T00:44:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 1 -- start start 2023-09-30T00:44:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 10 monclient: init 2023-09-30T00:44:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:37.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 0x7fce3012f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 10 monclient(hunting): picked mon.noname-c con 0x7fce30098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 0x7fce3009cc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.960+0000 7fce36de2700 10 monclient(hunting): picked mon.noname-a con 0x7fce30132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce30137b60 0x7fce301369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 10 monclient(hunting): picked mon.noname-b con 0x7fce30137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce3009d8a0 con 0x7fce30132d00 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce300a1500 con 0x7fce30137b60 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:37.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce301300f0 con 0x7fce30098670 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce36de2700 10 monclient(hunting): authenticate will time out at 1274.258027s 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 0x7fce3012f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 0x7fce3009cc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 0x7fce3012f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41474/0 (socket says 172.21.15.6:41474) 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 0x7fce3009cc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49840/0 (socket says 172.21.15.6:49840) 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 1 -- 172.21.15.6:0/741484130 learned_addr learned my addr 172.21.15.6:0/741484130 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): get_auth_request con 0x7fce30098670 auth_method 0 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:37.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): get_auth_request con 0x7fce30132d00 auth_method 0 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:37.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce3537f700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce30137b60 0x7fce301369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce3537f700 10 monclient(hunting): get_auth_request con 0x7fce30137b60 auth_method 0 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce3537f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce3537f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce3537f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient(hunting): handle_auth_done global_id 4344 payload 824 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 -- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 msgr2=0x7fce3012f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 0x7fce3012f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 -- 172.21.15.6:0/741484130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce30137b60 msgr2=0x7fce301369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce30137b60 0x7fce301369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 monclient: found mon.noname-a 2023-09-30T00:44:37.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce2ffff700 1 -- 172.21.15.6:0/741484130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce28009a20 con 0x7fce30132d00 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.961+0000 7fce34b7e700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 0x7fce3012f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2ffff700 1 -- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 msgr2=0x7fce3009cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2ffff700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 0x7fce3009cc60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fce2400ea50 tx=0x7fce240091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce240095d0 con 0x7fce30132d00 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 10 monclient: dump: 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:37.977 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce2400ca00 con 0x7fce30132d00 2023-09-30T00:44:37.978 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.963+0000 7fce36de2700 5 monclient: authenticate success, global_id 4344 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce36de2700 10 monclient: _renew_subs 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce36de2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3009dca0 con 0x7fce30132d00 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce2cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce2400cdc0 con 0x7fce30132d00 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce2cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce2cff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce2cff9700 10 monclient: dump: 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.979 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce2effd700 4 set_mon_vals no callback set 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce36de2700 10 monclient: _renew_subs 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce36de2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.964+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce30136f30 con 0x7fce30132d00 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.965+0000 7fce36de2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:37.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.965+0000 7fce36de2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.965+0000 7fce36de2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.965+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce30072b00 con 0x7fce30132d00 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.965+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fce24013680 con 0x7fce30132d00 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.966+0000 7fce2cff9700 1 --2- 172.21.15.6:0/741484130 >> v2:172.21.15.22:6812/142603 conn(0x7fce1c0357a0 0x7fce1c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.966+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fce240481f0 con 0x7fce30132d00 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.966+0000 7fce34b7e700 1 --2- 172.21.15.6:0/741484130 >> v2:172.21.15.22:6812/142603 conn(0x7fce1c0357a0 0x7fce1c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.966+0000 7fce34b7e700 10 monclient: get_auth_request con 0x7fce1c0357a0 auth_method 0 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.968+0000 7fce34b7e700 1 --2- 172.21.15.6:0/741484130 >> v2:172.21.15.22:6812/142603 conn(0x7fce1c0357a0 0x7fce1c037c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce200052e0 tx=0x7fce20004a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.970+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce24013930 con 0x7fce30132d00 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.970+0000 7fce2cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:37.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:37.970+0000 7fce2cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.219+0000 7fce36de2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.219+0000 7fce36de2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.219+0000 7fce36de2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.219+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce3009dc40 con 0x7fce30132d00 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.220+0000 7fce2cff9700 1 -- 172.21.15.6:0/741484130 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fce24021020 con 0x7fce30132d00 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.220+0000 7fce2cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.220+0000 7fce2cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:38.220 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:38.221 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 >> v2:172.21.15.22:6812/142603 conn(0x7fce1c0357a0 msgr2=0x7fce1c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 --2- 172.21.15.6:0/741484130 >> v2:172.21.15.22:6812/142603 conn(0x7fce1c0357a0 0x7fce1c037c50 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fce200052e0 tx=0x7fce20004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 10 monclient: shutdown 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 msgr2=0x7fce3009cc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 0x7fce3009cc60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fce2400ea50 tx=0x7fce240091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 shutdown_connections 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 --2- 172.21.15.6:0/741484130 >> v2:172.21.15.22:6812/142603 conn(0x7fce1c0357a0 0x7fce1c037c50 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce30137b60 0x7fce301369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce30098670 0x7fce3012f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 --2- 172.21.15.6:0/741484130 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce30132d00 0x7fce3009cc60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 >> 172.21.15.6:0/741484130 conn(0x7fce300a5480 msgr2=0x7fce30130510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 shutdown_connections 2023-09-30T00:44:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:38.222+0000 7fce36de2700 1 -- 172.21.15.6:0/741484130 wait complete. 2023-09-30T00:44:41.236 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:41.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.375+0000 7f97b8c93700 10 monclient: get_monmap_and_config 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.375+0000 7f97b8c93700 10 monclient: build_initial_monmap 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.375+0000 7f97b8c93700 10 monclient: monmap: 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:41.375235+0000 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:41.375235+0000 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:41.375 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:41.376 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:41.376 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:41.376 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:41.376 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.375+0000 7f97b8c93700 1 Processor -- start 2023-09-30T00:44:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.375+0000 7f97b8c93700 1 -- start start 2023-09-30T00:44:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.375+0000 7f97b8c93700 10 monclient: init 2023-09-30T00:44:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:41.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f97b4098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 0x7f97b4137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): picked mon.noname-b con 0x7f97b4132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f97b4137b60 0x7f97b4139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): picked mon.noname-c con 0x7f97b4137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97b40a1000 con 0x7f97b4098670 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97b4099150 con 0x7f97b4137b60 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40a0a50 con 0x7f97b4132d00 2023-09-30T00:44:41.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b8c93700 10 monclient(hunting): authenticate will time out at 1277.673414s 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 0x7f97b4137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 0x7f97b4137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41490/0 (socket says 172.21.15.6:41490) 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 -- 172.21.15.6:0/1650075962 learned_addr learned my addr 172.21.15.6:0/1650075962 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): get_auth_request con 0x7f97b4132d00 auth_method 0 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 1 --2- 172.21.15.6:0/1650075962 >> v2:172.21.15.22:3300/0 conn(0x7f97b4137b60 0x7f97b4139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): get_auth_request con 0x7f97b4137b60 auth_method 0 2023-09-30T00:44:41.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:41.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_done global_id 4340 payload 274 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 -- 172.21.15.6:0/1650075962 >> v2:172.21.15.22:3300/0 conn(0x7f97b4137b60 msgr2=0x7f97b4139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 --2- 172.21.15.6:0/1650075962 >> v2:172.21.15.22:3300/0 conn(0x7f97b4137b60 0x7f97b4139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 -- 172.21.15.6:0/1650075962 >> v1:172.21.15.22:6789/0 conn(0x7f97b4098670 legacy=0x7f97b4098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 monclient: found mon.noname-b 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.376+0000 7f97b1d9b700 1 -- 172.21.15.6:0/1650075962 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97b40a0730 con 0x7f97b4132d00 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b1d9b700 1 -- 172.21.15.6:0/1650075962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 msgr2=0x7f97b4137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b1d9b700 1 --2- 172.21.15.6:0/1650075962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 0x7f97b4137620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f979c0099b0 tx=0x7f979c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2e162c7e2b2e08fe server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 1 -- 172.21.15.6:0/1650075962 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c009ea0 con 0x7f97b4132d00 2023-09-30T00:44:41.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: dump: 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:41.385 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: _finish_auth 0 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 1 -- 172.21.15.6:0/1650075962 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979c00ba30 con 0x7f97b4132d00 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 5 monclient: authenticate success, global_id 4340 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 1 -- 172.21.15.6:0/1650075962 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f979c00bdf0 con 0x7f97b4132d00 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 4 set_mon_vals no callback set 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:41.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b159a700 10 monclient: dump: 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 10 monclient: shutdown 2023-09-30T00:44:41.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 1 -- 172.21.15.6:0/1650075962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 msgr2=0x7f97b4137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 1 --2- 172.21.15.6:0/1650075962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 0x7f97b4137620 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f979c0099b0 tx=0x7f979c009250 comp rx=0 tx=0).stop 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 1 -- 172.21.15.6:0/1650075962 shutdown_connections 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 1 --2- 172.21.15.6:0/1650075962 >> v2:172.21.15.22:3300/0 conn(0x7f97b4137b60 0x7f97b4139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 1 --2- 172.21.15.6:0/1650075962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4132d00 0x7f97b4137620 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.379+0000 7f97b8c93700 1 -- 172.21.15.6:0/1650075962 >> 172.21.15.6:0/1650075962 conn(0x7f97b40a5480 msgr2=0x7f97b40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 1 -- 172.21.15.6:0/1650075962 shutdown_connections 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 1 -- 172.21.15.6:0/1650075962 wait complete. 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 10 monclient: build_initial_monmap 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 10 monclient: monmap: 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:41.388 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 1 Processor -- start 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 1 -- start start 2023-09-30T00:44:41.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.380+0000 7f97b8c93700 10 monclient: init 2023-09-30T00:44:41.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:41.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97b4098670 0x7f97b409b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): picked mon.noname-a con 0x7f97b4098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97b4132d00 0x7f97b409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): picked mon.noname-b con 0x7f97b4132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 0x7f97b409cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): picked mon.noname-c con 0x7f97b4137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b40a1500 con 0x7f97b4098670 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:41.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4137810 con 0x7f97b4132d00 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f97b4136350 con 0x7f97b4137b60 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b8c93700 10 monclient(hunting): authenticate will time out at 1277.678351s 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 0x7f97b409cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97b4098670 0x7f97b409b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 0x7f97b409cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41502/0 (socket says 172.21.15.6:41502) 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 -- 172.21.15.6:0/3950399723 learned_addr learned my addr 172.21.15.6:0/3950399723 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97b4098670 0x7f97b409b9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49842/0 (socket says 172.21.15.6:49842) 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): get_auth_request con 0x7f97b4137b60 auth_method 0 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:41.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): get_auth_request con 0x7f97b4098670 auth_method 0 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:41.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97b4132d00 0x7f97b409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): get_auth_request con 0x7f97b4132d00 auth_method 0 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b1d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient(hunting): handle_auth_done global_id 4343 payload 824 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 -- 172.21.15.6:0/3950399723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97b4132d00 msgr2=0x7f97b409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97b4132d00 0x7f97b409c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 -- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97b4098670 msgr2=0x7f97b409b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:41.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97b4098670 0x7f97b409b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 monclient: found mon.noname-c 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.381+0000 7f97b2d9d700 1 -- 172.21.15.6:0/3950399723 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c0094f0 con 0x7f97b4137b60 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b2d9d700 1 -- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 msgr2=0x7f97b409cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b2d9d700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 0x7f97b409cce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00ea20 tx=0x7f97ac0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac009650 con 0x7f97b4137b60 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:41.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: dump: 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:41.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 5 monclient: authenticate success, global_id 4343 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f97ac00ca00 con 0x7f97b4137b60 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 10 monclient: _renew_subs 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97ac00cdc0 con 0x7f97b4137b60 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97b4133b10 con 0x7f97b4137b60 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b0d99700 4 set_mon_vals no callback set 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97aa7fc700 10 monclient: dump: 2023-09-30T00:44:41.400 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 10 monclient: _renew_subs 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:41.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.384+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97b409e5c0 con 0x7f97b4137b60 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f97ac013680 con 0x7f97b4137b60 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97b8c93700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97b8c93700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97b8c93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97b4072b00 con 0x7f97b4137b60 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97aa7fc700 1 --2- 172.21.15.6:0/3950399723 >> v2:172.21.15.22:6812/142603 conn(0x7f97940357a0 0x7f9794037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f97ac047bb0 con 0x7f97b4137b60 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97b259c700 1 --2- 172.21.15.6:0/3950399723 >> v2:172.21.15.22:6812/142603 conn(0x7f97940357a0 0x7f9794037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.385+0000 7f97b259c700 10 monclient: get_auth_request con 0x7f97940357a0 auth_method 0 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.388+0000 7f97b259c700 1 --2- 172.21.15.6:0/3950399723 >> v2:172.21.15.22:6812/142603 conn(0x7f97940357a0 0x7f9794037c50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a40026a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.390+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f97ac0123d0 con 0x7f97b4137b60 2023-09-30T00:44:41.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.390+0000 7f97aa7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:41.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.390+0000 7f97aa7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97b8c93700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97b8c93700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97b8c93700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f97b4063920 con 0x7f97b4137b60 2023-09-30T00:44:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97aa7fc700 1 -- 172.21.15.6:0/3950399723 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f97ac021070 con 0x7f97b4137b60 2023-09-30T00:44:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97aa7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.635+0000 7f97aa7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:41.636 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:41.636 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 >> v2:172.21.15.22:6812/142603 conn(0x7f97940357a0 msgr2=0x7f9794037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 --2- 172.21.15.6:0/3950399723 >> v2:172.21.15.22:6812/142603 conn(0x7f97940357a0 0x7f9794037c50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f97a4007fd0 tx=0x7f97a40026a0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 10 monclient: shutdown 2023-09-30T00:44:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 msgr2=0x7f97b409cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 0x7f97b409cce0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f97ac00ea20 tx=0x7f97ac0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 shutdown_connections 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 --2- 172.21.15.6:0/3950399723 >> v2:172.21.15.22:6812/142603 conn(0x7f97940357a0 0x7f9794037c50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f97b4137b60 0x7f97b409cce0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f97b4098670 0x7f97b409b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 --2- 172.21.15.6:0/3950399723 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f97b4132d00 0x7f97b409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 >> 172.21.15.6:0/3950399723 conn(0x7f97b40a5480 msgr2=0x7f97b4131b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 shutdown_connections 2023-09-30T00:44:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:41.638+0000 7f97b8c93700 1 -- 172.21.15.6:0/3950399723 wait complete. 2023-09-30T00:44:44.652 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.790+0000 7f443cb46700 10 monclient: get_monmap_and_config 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.790+0000 7f443cb46700 10 monclient: build_initial_monmap 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.790+0000 7f443cb46700 10 monclient: monmap: 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:44.790290+0000 2023-09-30T00:44:44.790 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:44.790290+0000 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.790+0000 7f443cb46700 1 Processor -- start 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.790+0000 7f443cb46700 1 -- start start 2023-09-30T00:44:44.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.790+0000 7f443cb46700 10 monclient: init 2023-09-30T00:44:44.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:44.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-09-30T00:44:44.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4438098670 0x7f4438132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): picked mon.noname-c con 0x7f4438098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4438098a40 0x7f443812ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): picked mon.noname-a con 0x7f4438098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4438098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44380a1000 con 0x7f4438098e90 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44380a0a50 con 0x7f4438098a40 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44380a0730 con 0x7f4438098670 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:44.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443cb46700 10 monclient(hunting): authenticate will time out at 1281.088487s 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4438098a40 0x7f443812ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4438098a40 0x7f443812ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49856/0 (socket says 172.21.15.6:49856) 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 -- 172.21.15.6:0/3229725062 learned_addr learned my addr 172.21.15.6:0/3229725062 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): get_auth_request con 0x7f4438098a40 auth_method 0 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:44.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443659c700 1 --2- 172.21.15.6:0/3229725062 >> v2:172.21.15.22:3300/0 conn(0x7f4438098670 0x7f4438132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443659c700 10 monclient(hunting): get_auth_request con 0x7f4438098670 auth_method 0 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f443659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient(hunting): handle_auth_done global_id 4350 payload 274 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 -- 172.21.15.6:0/3229725062 >> v2:172.21.15.22:3300/0 conn(0x7f4438098670 msgr2=0x7f4438132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 --2- 172.21.15.6:0/3229725062 >> v2:172.21.15.22:3300/0 conn(0x7f4438098670 0x7f4438132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 -- 172.21.15.6:0/3229725062 >> v1:172.21.15.22:6789/0 conn(0x7f4438098e90 legacy=0x7f443812f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:44.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 monclient: found mon.noname-a 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.791+0000 7f4435d9b700 1 -- 172.21.15.6:0/3229725062 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4438068df0 con 0x7f4438098a40 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f4435d9b700 1 -- 172.21.15.6:0/3229725062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098a40 msgr2=0x7f443812ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f4435d9b700 1 --2- 172.21.15.6:0/3229725062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098a40 0x7f443812ede0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f443809fe60 tx=0x7f442c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=163a58fb4866c3a7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 1 -- 172.21.15.6:0/3229725062 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c00ba30 con 0x7f4438098a40 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: dump: 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:44.799 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: _finish_auth 0 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 1 -- 172.21.15.6:0/3229725062 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f442c00bb90 con 0x7f4438098a40 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 5 monclient: authenticate success, global_id 4350 2023-09-30T00:44:44.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 1 -- 172.21.15.6:0/3229725062 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f442c011420 con 0x7f4438098a40 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 4 set_mon_vals no callback set 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443559a700 10 monclient: dump: 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:44.801 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 10 monclient: shutdown 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 1 -- 172.21.15.6:0/3229725062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098a40 msgr2=0x7f443812ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.794+0000 7f443cb46700 1 --2- 172.21.15.6:0/3229725062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098a40 0x7f443812ede0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f443809fe60 tx=0x7f442c009250 comp rx=0 tx=0).stop 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 1 -- 172.21.15.6:0/3229725062 shutdown_connections 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 1 --2- 172.21.15.6:0/3229725062 >> v2:172.21.15.22:3300/0 conn(0x7f4438098670 0x7f4438132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:44.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 1 --2- 172.21.15.6:0/3229725062 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098a40 0x7f443812ede0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 1 -- 172.21.15.6:0/3229725062 >> 172.21.15.6:0/3229725062 conn(0x7f44380a5480 msgr2=0x7f44380a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 1 -- 172.21.15.6:0/3229725062 shutdown_connections 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 1 -- 172.21.15.6:0/3229725062 wait complete. 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 10 monclient: build_initial_monmap 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.795+0000 7f443cb46700 10 monclient: monmap: 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:44.803 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:44.804 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 Processor -- start 2023-09-30T00:44:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 -- start start 2023-09-30T00:44:44.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient: init 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 0x7f4438141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): picked mon.noname-c con 0x7f4438098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4438098a40 0x7f443812c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): picked mon.noname-b con 0x7f4438098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098e90 0x7f443809cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): picked mon.noname-a con 0x7f4438098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:44.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443811ccd0 con 0x7f4438098e90 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f443811c430 con 0x7f4438098a40 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f443811c710 con 0x7f4438098670 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443cb46700 10 monclient(hunting): authenticate will time out at 1281.093601s 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 0x7f4438141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098e90 0x7f443809cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 0x7f4438141930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41512/0 (socket says 172.21.15.6:41512) 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 1 -- 172.21.15.6:0/1636865999 learned_addr learned my addr 172.21.15.6:0/1636865999 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): get_auth_request con 0x7f4438098670 auth_method 0 2023-09-30T00:44:44.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): get_auth_request con 0x7f4438098e90 auth_method 0 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4438098a40 0x7f443812c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): get_auth_request con 0x7f4438098a40 auth_method 0 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:44.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f443659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4436d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.796+0000 7f4435d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 10 monclient(hunting): handle_auth_done global_id 4346 payload 824 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 1 -- 172.21.15.6:0/1636865999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4438098a40 msgr2=0x7f443812c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4438098a40 0x7f443812c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:44.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 1 -- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098e90 msgr2=0x7f443809cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098e90 0x7f443809cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 10 monclient: _finish_hunting 0 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 1 monclient: found mon.noname-c 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.797+0000 7f443659c700 1 -- 172.21.15.6:0/1636865999 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c0094f0 con 0x7f4438098670 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443659c700 1 -- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 msgr2=0x7f4438141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443659c700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 0x7f4438141930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f442800d890 tx=0x7f4428008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428012800 con 0x7f4438098670 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:44.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: dump: 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:44.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4428012960 con 0x7f4438098670 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 5 monclient: authenticate success, global_id 4346 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 10 monclient: _renew_subs 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f443811c9f0 con 0x7f4438098670 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4428012d00 con 0x7f4438098670 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f44267fc700 10 monclient: dump: 2023-09-30T00:44:44.814 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f4434d99700 4 set_mon_vals no callback set 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 10 monclient: _renew_subs 2023-09-30T00:44:44.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.799+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f443809f370 con 0x7f4438098670 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4428014640 con 0x7f4438098670 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f443cb46700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f443cb46700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f443cb46700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4438098320 con 0x7f4438098670 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f44267fc700 1 --2- 172.21.15.6:0/1636865999 >> v2:172.21.15.22:6812/142603 conn(0x7f44180357a0 0x7f4418037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4428047f20 con 0x7f4438098670 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f4435d9b700 1 --2- 172.21.15.6:0/1636865999 >> v2:172.21.15.22:6812/142603 conn(0x7f44180357a0 0x7f4418037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.800+0000 7f4435d9b700 10 monclient: get_auth_request con 0x7f44180357a0 auth_method 0 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.803+0000 7f4435d9b700 1 --2- 172.21.15.6:0/1636865999 >> v2:172.21.15.22:6812/142603 conn(0x7f44180357a0 0x7f4418037c50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f443809fe60 tx=0x7f442c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:44.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.805+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4428012e60 con 0x7f4438098670 2023-09-30T00:44:44.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.805+0000 7f44267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:44.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:44.805+0000 7f44267fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.055+0000 7f443cb46700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.055+0000 7f443cb46700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.055+0000 7f443cb46700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.055+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f443809e6f0 con 0x7f4438098670 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.056+0000 7f44267fc700 1 -- 172.21.15.6:0/1636865999 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f442801c360 con 0x7f4438098670 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.056+0000 7f44267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.056+0000 7f44267fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:45.056 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 >> v2:172.21.15.22:6812/142603 conn(0x7f44180357a0 msgr2=0x7f4418037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 --2- 172.21.15.6:0/1636865999 >> v2:172.21.15.22:6812/142603 conn(0x7f44180357a0 0x7f4418037c50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f443809fe60 tx=0x7f442c016000 comp rx=0 tx=0).stop 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 10 monclient: shutdown 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 msgr2=0x7f4438141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 0x7f4438141930 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f442800d890 tx=0x7f4428008040 comp rx=0 tx=0).stop 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 shutdown_connections 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 --2- 172.21.15.6:0/1636865999 >> v2:172.21.15.22:6812/142603 conn(0x7f44180357a0 0x7f4418037c50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4438098e90 0x7f443809cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4438098670 0x7f4438141930 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 --2- 172.21.15.6:0/1636865999 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4438098a40 0x7f443812c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 >> 172.21.15.6:0/1636865999 conn(0x7f44380a5480 msgr2=0x7f44381308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 shutdown_connections 2023-09-30T00:44:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:45.058+0000 7f443cb46700 1 -- 172.21.15.6:0/1636865999 wait complete. 2023-09-30T00:44:48.072 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:48.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.210+0000 7fa407089700 10 monclient: get_monmap_and_config 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.210+0000 7fa407089700 10 monclient: build_initial_monmap 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.210+0000 7fa407089700 10 monclient: monmap: 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:48.211280+0000 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:48.211280+0000 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:48.211 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:48.212 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:48.212 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.210+0000 7fa407089700 1 Processor -- start 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 1 -- start start 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient: init 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient: _add_conns ranks=[1,4,3,0,2] 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa4000675f0 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:48.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa400097a90 0x7fa4001373e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): picked mon.noname-c con 0x7fa400097a90 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa400097eb0 0x7fa400139930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): picked mon.noname-a con 0x7fa400097eb0 addr v2:172.21.15.6:3300/0 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa40006a3a0 con 0x7fa4000675f0 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa40009eac0 con 0x7fa400097eb0 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4000965d0 con 0x7fa400097a90 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa407089700 10 monclient(hunting): authenticate will time out at 1284.509667s 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa405626700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa400097eb0 0x7fa400139930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:48.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa404e25700 1 --1- >> v1:172.21.15.22:6789/0 conn(0x7fa4000675f0 0x7fa400098580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.22:6789/0 says I am v1:172.21.15.6:47988/0 (socket says 172.21.15.6:47988) 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa405626700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa400097eb0 0x7fa400139930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42098/0 (socket says 172.21.15.6:42098) 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa404e25700 1 -- 172.21.15.6:0/398629036 learned_addr learned my addr 172.21.15.6:0/398629036 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa405626700 10 monclient(hunting): get_auth_request con 0x7fa400097eb0 auth_method 0 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa3fffff700 1 --2- 172.21.15.6:0/398629036 >> v2:172.21.15.22:3300/0 conn(0x7fa400097a90 0x7fa4001373e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa405626700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa405626700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa405626700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa3fffff700 10 monclient(hunting): get_auth_request con 0x7fa400097a90 auth_method 0 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa3fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa3fffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.211+0000 7fa3fffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:48.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa3fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa3fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa3fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 10 monclient(hunting): handle_auth_done global_id 4356 payload 274 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 1 -- 172.21.15.6:0/398629036 >> v2:172.21.15.22:3300/0 conn(0x7fa400097a90 msgr2=0x7fa4001373e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 1 --2- 172.21.15.6:0/398629036 >> v2:172.21.15.22:3300/0 conn(0x7fa400097a90 0x7fa4001373e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 1 -- 172.21.15.6:0/398629036 >> v1:172.21.15.22:6789/0 conn(0x7fa4000675f0 legacy=0x7fa400098580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:48.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 10 monclient: _finish_hunting 0 2023-09-30T00:44:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 1 monclient: found mon.noname-a 2023-09-30T00:44:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:48.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.212+0000 7fa405626700 1 -- 172.21.15.6:0/398629036 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4000a3140 con 0x7fa400097eb0 2023-09-30T00:44:48.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa405626700 1 -- 172.21.15.6:0/398629036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097eb0 msgr2=0x7fa400139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa405626700 1 --2- 172.21.15.6:0/398629036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097eb0 0x7fa400139930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa40006ac20 tx=0x7fa3f8009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97a1e533ad1885d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 1 -- 172.21.15.6:0/398629036 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800e070 con 0x7fa400097eb0 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: dump: 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:48.225 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa407089700 5 monclient: authenticate success, global_id 4356 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa407089700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 1 -- 172.21.15.6:0/398629036 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3f800bd30 con 0x7fa400097eb0 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:48.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa407089700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 1 -- 172.21.15.6:0/398629036 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3f800bea0 con 0x7fa400097eb0 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa407089700 4 set_mon_vals no callback set 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.214+0000 7fa3ff7fe700 10 monclient: dump: 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:48.227 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 10 monclient: shutdown 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 -- 172.21.15.6:0/398629036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097eb0 msgr2=0x7fa400139930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 --2- 172.21.15.6:0/398629036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097eb0 0x7fa400139930 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa40006ac20 tx=0x7fa3f8009780 comp rx=0 tx=0).stop 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 -- 172.21.15.6:0/398629036 shutdown_connections 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 --2- 172.21.15.6:0/398629036 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097eb0 0x7fa400139930 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 --2- 172.21.15.6:0/398629036 >> v2:172.21.15.22:3300/0 conn(0x7fa400097a90 0x7fa4001373e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 -- 172.21.15.6:0/398629036 >> 172.21.15.6:0/398629036 conn(0x7fa4000a21d0 msgr2=0x7fa4000a25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 -- 172.21.15.6:0/398629036 shutdown_connections 2023-09-30T00:44:48.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.215+0000 7fa407089700 1 -- 172.21.15.6:0/398629036 wait complete. 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.216+0000 7fa407089700 10 monclient: build_initial_monmap 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.216+0000 7fa407089700 10 monclient: monmap: 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:48.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.216+0000 7fa407089700 1 Processor -- start 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.216+0000 7fa407089700 1 -- start start 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.216+0000 7fa407089700 10 monclient: init 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4000675f0 0x7fa400094f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): picked mon.noname-b con 0x7fa4000675f0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 0x7fa40006cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): picked mon.noname-a con 0x7fa400097a90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa400097eb0 0x7fa40006d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): picked mon.noname-c con 0x7fa400097eb0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:48.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4000a0660 con 0x7fa400097a90 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa40008cfa0 con 0x7fa4000675f0 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa40006f230 con 0x7fa400097eb0 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa407089700 10 monclient(hunting): authenticate will time out at 1284.515158s 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 0x7fa40006cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 0x7fa40006cee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42110/0 (socket says 172.21.15.6:42110) 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 -- 172.21.15.6:0/1346520089 learned_addr learned my addr 172.21.15.6:0/1346520089 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): get_auth_request con 0x7fa400097a90 auth_method 0 2023-09-30T00:44:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa400097eb0 0x7fa40006d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): get_auth_request con 0x7fa400097eb0 auth_method 0 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa405626700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa404e25700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4000675f0 0x7fa400094f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa404e25700 10 monclient(hunting): get_auth_request con 0x7fa4000675f0 auth_method 0 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa404e25700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa404e25700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa404e25700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient(hunting): handle_auth_done global_id 4359 payload 824 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 -- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa400097eb0 msgr2=0x7fa40006d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa400097eb0 0x7fa40006d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 -- 172.21.15.6:0/1346520089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4000675f0 msgr2=0x7fa400094f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4000675f0 0x7fa400094f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 monclient: found mon.noname-a 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.217+0000 7fa3fffff700 1 -- 172.21.15.6:0/1346520089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3f8009a20 con 0x7fa400097a90 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.219+0000 7fa3fffff700 1 -- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 msgr2=0x7fa40006cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.219+0000 7fa3fffff700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 0x7fa40006cee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00ec80 tx=0x7fa3ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec00ca60 con 0x7fa400097a90 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: dump: 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:48.234 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 5 monclient: authenticate success, global_id 4359 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 10 monclient: _renew_subs 2023-09-30T00:44:48.235 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3ec00cbc0 con 0x7fa400097a90 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa40006f700 con 0x7fa400097a90 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3ec013470 con 0x7fa400097a90 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3fcff9700 10 monclient: dump: 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:48.236 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa3feffd700 4 set_mon_vals no callback set 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 10 monclient: _renew_subs 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.220+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa40008dc60 con 0x7fa400097a90 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa3ec0137e0 con 0x7fa400097a90 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa407089700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa407089700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa407089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa400072b00 con 0x7fa400097a90 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa3fcff9700 1 --2- 172.21.15.6:0/1346520089 >> v2:172.21.15.22:6812/142603 conn(0x7fa3e40357a0 0x7fa3e4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa3ec047cb0 con 0x7fa400097a90 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa404e25700 1 --2- 172.21.15.6:0/1346520089 >> v2:172.21.15.22:6812/142603 conn(0x7fa3e40357a0 0x7fa3e4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.221+0000 7fa404e25700 10 monclient: get_auth_request con 0x7fa3e40357a0 auth_method 0 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.224+0000 7fa404e25700 1 --2- 172.21.15.6:0/1346520089 >> v2:172.21.15.22:6812/142603 conn(0x7fa3e40357a0 0x7fa3e4037c50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4007fd0 tx=0x7fa3f4007ee0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.226+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa3ec0236b0 con 0x7fa400097a90 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.226+0000 7fa3fcff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.226+0000 7fa3fcff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa407089700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa407089700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa407089700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa4000684e0 con 0x7fa400097a90 2023-09-30T00:44:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa3fcff9700 1 -- 172.21.15.6:0/1346520089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa3ec0232f0 con 0x7fa400097a90 2023-09-30T00:44:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa3fcff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.474+0000 7fa3fcff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:48.476 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:48.476 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 >> v2:172.21.15.22:6812/142603 conn(0x7fa3e40357a0 msgr2=0x7fa3e4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 --2- 172.21.15.6:0/1346520089 >> v2:172.21.15.22:6812/142603 conn(0x7fa3e40357a0 0x7fa3e4037c50 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4007fd0 tx=0x7fa3f4007ee0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 10 monclient: shutdown 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 msgr2=0x7fa40006cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 0x7fa40006cee0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fa3ec00ec80 tx=0x7fa3ec0091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 shutdown_connections 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 --2- 172.21.15.6:0/1346520089 >> v2:172.21.15.22:6812/142603 conn(0x7fa3e40357a0 0x7fa3e4037c50 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa400097eb0 0x7fa40006d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa4000675f0 0x7fa400094f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 --2- 172.21.15.6:0/1346520089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa400097a90 0x7fa40006cee0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 >> 172.21.15.6:0/1346520089 conn(0x7fa4000a21d0 msgr2=0x7fa40009ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 shutdown_connections 2023-09-30T00:44:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:48.477+0000 7fa407089700 1 -- 172.21.15.6:0/1346520089 wait complete. 2023-09-30T00:44:51.492 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:51.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.630+0000 7f337f21d700 10 monclient: get_monmap_and_config 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.630+0000 7f337f21d700 10 monclient: build_initial_monmap 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.630+0000 7f337f21d700 10 monclient: monmap: 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:51.632100+0000 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:51.632100+0000 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:51.632 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:51.633 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:51.633 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:51.633 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.631+0000 7f337f21d700 1 Processor -- start 2023-09-30T00:44:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.631+0000 7f337f21d700 1 -- start start 2023-09-30T00:44:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.631+0000 7f337f21d700 10 monclient: init 2023-09-30T00:44:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:51.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 0x7f3378132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): picked mon.noname-c con 0x7f3378098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3378098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337812f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): picked mon.noname-b con 0x7f3378098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33780a1000 con 0x7f3378098a40 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33780a0a50 con 0x7f3378098670 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33780a0730 con 0x7f3378098e90 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:51.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337f21d700 10 monclient(hunting): authenticate will time out at 1287.930680s 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337812f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f3377fff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3378098a40 0x7f337812ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35338/0 (socket says 172.21.15.6:35338) 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f3377fff700 1 -- 172.21.15.6:0/2721620954 learned_addr learned my addr 172.21.15.6:0/2721620954 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): get_auth_request con 0x7f3378098e90 auth_method 0 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337d7ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.632+0000 7f337cfb9700 1 -- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 msgr2=0x7f3378132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2023-09-30T00:44:51.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337cfb9700 1 -- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 msgr2=0x7f3378132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337cfb9700 1 --2- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 0x7f3378132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner read peer banner failed r=-1 ((1) Operation not permitted) 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337cfb9700 1 --2- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 0x7f3378132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 10 monclient(hunting): handle_auth_done global_id 4352 payload 274 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 1 -- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 msgr2=0x7f3378132d00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 1 --2- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 0x7f3378132d00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 1 -- 172.21.15.6:0/2721620954 >> v1:172.21.15.6:6789/0 conn(0x7f3378098a40 legacy=0x7f337812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 10 monclient: _finish_hunting 0 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 1 monclient: found mon.noname-b 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:51.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.633+0000 7f337d7ba700 1 -- 172.21.15.6:0/2721620954 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3378068df0 con 0x7f3378098e90 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f337d7ba700 1 -- 172.21.15.6:0/2721620954 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 msgr2=0x7f337812f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f337d7ba700 1 --2- 172.21.15.6:0/2721620954 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337812f4f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3370009e60 tx=0x7f3370009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66c38c1dc53b4aae server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 1 -- 172.21.15.6:0/2721620954 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f337000e070 con 0x7f3378098e90 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: dump: 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:51.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 1 -- 172.21.15.6:0/2721620954 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3370012430 con 0x7f3378098e90 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f337f21d700 5 monclient: authenticate success, global_id 4352 2023-09-30T00:44:51.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f337f21d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f337f21d700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 1 -- 172.21.15.6:0/2721620954 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f33700127d0 con 0x7f3378098e90 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f337f21d700 4 set_mon_vals no callback set 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.635+0000 7f33777fe700 10 monclient: dump: 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.646 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 10 monclient: shutdown 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 -- 172.21.15.6:0/2721620954 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 msgr2=0x7f337812f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 --2- 172.21.15.6:0/2721620954 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337812f4f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f3370009e60 tx=0x7f3370009780 comp rx=0 tx=0).stop 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 -- 172.21.15.6:0/2721620954 shutdown_connections 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 --2- 172.21.15.6:0/2721620954 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337812f4f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 --2- 172.21.15.6:0/2721620954 >> v2:172.21.15.22:3300/0 conn(0x7f3378098670 0x7f3378132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 -- 172.21.15.6:0/2721620954 >> 172.21.15.6:0/2721620954 conn(0x7f33780a5480 msgr2=0x7f33780a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 -- 172.21.15.6:0/2721620954 shutdown_connections 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 1 -- 172.21.15.6:0/2721620954 wait complete. 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 10 monclient: build_initial_monmap 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.636+0000 7f337f21d700 10 monclient: monmap: 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:51.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 Processor -- start 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 -- start start 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient: init 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3378098670 0x7f337812d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): picked mon.noname-b con 0x7f3378098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 0x7f337812d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:51.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): picked mon.noname-a con 0x7f3378098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337809d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): picked mon.noname-c con 0x7f3378098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f337809cc60 con 0x7f3378098a40 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33780a1500 con 0x7f3378098670 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f337812edc0 con 0x7f3378098e90 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337f21d700 10 monclient(hunting): authenticate will time out at 1287.935783s 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 0x7f337812d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:51.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337d7ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337809d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 0x7f337812d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42120/0 (socket says 172.21.15.6:42120) 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337d7ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337809d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:60332/0 (socket says 172.21.15.6:60332) 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 1 -- 172.21.15.6:0/1182590162 learned_addr learned my addr 172.21.15.6:0/1182590162 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 10 monclient(hunting): get_auth_request con 0x7f3378098a40 auth_method 0 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f3377fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337d7ba700 10 monclient(hunting): get_auth_request con 0x7f3378098e90 auth_method 0 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337d7ba700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337d7ba700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:51.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.637+0000 7f337d7ba700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3378098670 0x7f337812d470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): get_auth_request con 0x7f3378098670 auth_method 0 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337d7ba700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337d7ba700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337d7ba700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:51.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337cfb9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 10 monclient(hunting): handle_auth_done global_id 4362 payload 824 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 1 -- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 msgr2=0x7f337809d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337809d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 1 -- 172.21.15.6:0/1182590162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3378098670 msgr2=0x7f337812d470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3378098670 0x7f337812d470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 10 monclient: _finish_hunting 0 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 1 monclient: found mon.noname-a 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f3377fff700 1 -- 172.21.15.6:0/1182590162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3370009a20 con 0x7f3378098a40 2023-09-30T00:44:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.638+0000 7f337d7ba700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337809d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3377fff700 1 -- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 msgr2=0x7f337812d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3377fff700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 0x7f337812d9b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f336000c5e0 tx=0x7f3360004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360005d40 con 0x7f3378098a40 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: dump: 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: _finish_auth 0 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33600070d0 con 0x7f3378098a40 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 5 monclient: authenticate success, global_id 4362 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 10 monclient: _renew_subs 2023-09-30T00:44:51.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3360007490 con 0x7f3378098a40 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33780a39f0 con 0x7f3378098a40 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3376ffd700 4 set_mon_vals no callback set 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f3374ff9700 10 monclient: dump: 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 10 monclient: _renew_subs 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.640+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f337812e8c0 con 0x7f3378098a40 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f336001b3b0 con 0x7f3378098a40 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f3374ff9700 1 --2- 172.21.15.6:0/1182590162 >> v2:172.21.15.22:6812/142603 conn(0x7f33580357a0 0x7f3358037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f337f21d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f337f21d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3360046dd0 con 0x7f3378098a40 2023-09-30T00:44:51.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f337f21d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:51.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.641+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3378072b00 con 0x7f3378098a40 2023-09-30T00:44:51.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.642+0000 7f337cfb9700 1 --2- 172.21.15.6:0/1182590162 >> v2:172.21.15.22:6812/142603 conn(0x7f33580357a0 0x7f3358037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:51.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.642+0000 7f337cfb9700 10 monclient: get_auth_request con 0x7f33580357a0 auth_method 0 2023-09-30T00:44:51.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.644+0000 7f337cfb9700 1 --2- 172.21.15.6:0/1182590162 >> v2:172.21.15.22:6812/142603 conn(0x7f33580357a0 0x7f3358037c50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3368005940 tx=0x7f33680091d0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:51.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.646+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3360017070 con 0x7f3378098a40 2023-09-30T00:44:51.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.646+0000 7f3374ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:51.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.646+0000 7f3374ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.895+0000 7f337f21d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.896+0000 7f337f21d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.896+0000 7f337f21d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.896+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f337812e010 con 0x7f3378098a40 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.896+0000 7f3374ff9700 1 -- 172.21.15.6:0/1182590162 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f336001a360 con 0x7f3378098a40 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.896+0000 7f3374ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.896+0000 7f3374ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:51.898 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:51.898 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 >> v2:172.21.15.22:6812/142603 conn(0x7f33580357a0 msgr2=0x7f3358037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 --2- 172.21.15.6:0/1182590162 >> v2:172.21.15.22:6812/142603 conn(0x7f33580357a0 0x7f3358037c50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f3368005940 tx=0x7f33680091d0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 10 monclient: shutdown 2023-09-30T00:44:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 msgr2=0x7f337812d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 0x7f337812d9b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f336000c5e0 tx=0x7f3360004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 shutdown_connections 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 --2- 172.21.15.6:0/1182590162 >> v2:172.21.15.22:6812/142603 conn(0x7f33580357a0 0x7f3358037c50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3378098e90 0x7f337809d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3378098670 0x7f337812d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 --2- 172.21.15.6:0/1182590162 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3378098a40 0x7f337812d9b0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 >> 172.21.15.6:0/1182590162 conn(0x7f33780a5480 msgr2=0x7f33780a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 shutdown_connections 2023-09-30T00:44:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:51.898+0000 7f337f21d700 1 -- 172.21.15.6:0/1182590162 wait complete. 2023-09-30T00:44:54.913 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:55.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.050+0000 7f59e02b2700 10 monclient: get_monmap_and_config 2023-09-30T00:44:55.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.050+0000 7f59e02b2700 10 monclient: build_initial_monmap 2023-09-30T00:44:55.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.050+0000 7f59e02b2700 10 monclient: monmap: 2023-09-30T00:44:55.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:55.051993+0000 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:55.051993+0000 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:55.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.051+0000 7f59e02b2700 1 Processor -- start 2023-09-30T00:44:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.051+0000 7f59e02b2700 1 -- start start 2023-09-30T00:44:55.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.051+0000 7f59e02b2700 10 monclient: init 2023-09-30T00:44:55.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient: _add_conns ranks=[0,2,4,3,1] 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f59d8098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 0x7f59d8137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): picked mon.noname-b con 0x7f59d8132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f59d8137b60 0x7f59d8139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): picked mon.noname-c con 0x7f59d8137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59d80a1000 con 0x7f59d8098670 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59d8099150 con 0x7f59d8137b60 2023-09-30T00:44:55.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80a0a50 con 0x7f59d8132d00 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59e02b2700 10 monclient(hunting): authenticate will time out at 1291.350222s 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 0x7f59d8137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de04e700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f59d8098670 0x7f59d8098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:35352/0 (socket says 172.21.15.6:35352) 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de04e700 1 -- 172.21.15.6:0/2897785350 learned_addr learned my addr 172.21.15.6:0/2897785350 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): get_auth_request con 0x7f59d8132d00 auth_method 0 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:55.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de84f700 1 --2- 172.21.15.6:0/2897785350 >> v2:172.21.15.22:3300/0 conn(0x7f59d8137b60 0x7f59d8139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de84f700 10 monclient(hunting): get_auth_request con 0x7f59d8137b60 auth_method 0 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de84f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de84f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de84f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient(hunting): handle_auth_done global_id 4358 payload 274 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 1 -- 172.21.15.6:0/2897785350 >> v2:172.21.15.22:3300/0 conn(0x7f59d8137b60 msgr2=0x7f59d8139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 1 --2- 172.21.15.6:0/2897785350 >> v2:172.21.15.22:3300/0 conn(0x7f59d8137b60 0x7f59d8139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 1 -- 172.21.15.6:0/2897785350 >> v1:172.21.15.6:6789/0 conn(0x7f59d8098670 legacy=0x7f59d8098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient: _finish_hunting 0 2023-09-30T00:44:55.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 1 monclient: found mon.noname-b 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59dd84d700 1 -- 172.21.15.6:0/2897785350 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59d80a0730 con 0x7f59d8132d00 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.052+0000 7f59de84f700 1 --2- 172.21.15.6:0/2897785350 >> v2:172.21.15.22:3300/0 conn(0x7f59d8137b60 0x7f59d8139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd84d700 1 -- 172.21.15.6:0/2897785350 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 msgr2=0x7f59d8137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd84d700 1 --2- 172.21.15.6:0/2897785350 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 0x7f59d8137620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f59c80098e0 tx=0x7f59c80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=747c1bd02e1d7b80 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 1 -- 172.21.15.6:0/2897785350 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c8009e80 con 0x7f59d8132d00 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:55.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: dump: 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: _finish_auth 0 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:55.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 1 -- 172.21.15.6:0/2897785350 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59c800ba30 con 0x7f59d8132d00 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 5 monclient: authenticate success, global_id 4358 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 1 -- 172.21.15.6:0/2897785350 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59c800bdb0 con 0x7f59d8132d00 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 4 set_mon_vals no callback set 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59dd04c700 10 monclient: dump: 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:55.068 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 10 monclient: shutdown 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 -- 172.21.15.6:0/2897785350 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 msgr2=0x7f59d8137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:55.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 --2- 172.21.15.6:0/2897785350 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 0x7f59d8137620 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f59c80098e0 tx=0x7f59c80091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 -- 172.21.15.6:0/2897785350 shutdown_connections 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 --2- 172.21.15.6:0/2897785350 >> v2:172.21.15.22:3300/0 conn(0x7f59d8137b60 0x7f59d8139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 --2- 172.21.15.6:0/2897785350 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8132d00 0x7f59d8137620 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 -- 172.21.15.6:0/2897785350 >> 172.21.15.6:0/2897785350 conn(0x7f59d80a5480 msgr2=0x7f59d80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 -- 172.21.15.6:0/2897785350 shutdown_connections 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.055+0000 7f59e02b2700 1 -- 172.21.15.6:0/2897785350 wait complete. 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.056+0000 7f59e02b2700 10 monclient: build_initial_monmap 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.056+0000 7f59e02b2700 10 monclient: monmap: 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:55.070 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.056+0000 7f59e02b2700 1 Processor -- start 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.056+0000 7f59e02b2700 1 -- start start 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.056+0000 7f59e02b2700 10 monclient: init 2023-09-30T00:44:55.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 0x7f59d812f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): picked mon.noname-c con 0x7f59d8098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59d8132d00 0x7f59d809c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): picked mon.noname-b con 0x7f59d8132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 0x7f59d809cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): picked mon.noname-a con 0x7f59d8137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d80a36b0 con 0x7f59d8137b60 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59d809f790 con 0x7f59d8132d00 2023-09-30T00:44:55.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59d809f4b0 con 0x7f59d8098670 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59e02b2700 10 monclient(hunting): authenticate will time out at 1291.355306s 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 0x7f59d812f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 0x7f59d809cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 0x7f59d812f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:60346/0 (socket says 172.21.15.6:60346) 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 1 -- 172.21.15.6:0/1561518233 learned_addr learned my addr 172.21.15.6:0/1561518233 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): get_auth_request con 0x7f59d8137b60 auth_method 0 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:55.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): get_auth_request con 0x7f59d8098670 auth_method 0 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59d8132d00 0x7f59d809c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): get_auth_request con 0x7f59d8132d00 auth_method 0 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59dd84d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient(hunting): handle_auth_done global_id 4365 payload 824 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 -- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 msgr2=0x7f59d812f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 0x7f59d812f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 -- 172.21.15.6:0/1561518233 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59d8132d00 msgr2=0x7f59d809c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59d8132d00 0x7f59d809c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient: _finish_hunting 0 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 monclient: found mon.noname-a 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de84f700 1 -- 172.21.15.6:0/1561518233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c8009420 con 0x7f59d8137b60 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.057+0000 7f59de04e700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 0x7f59d812f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59de84f700 1 -- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 msgr2=0x7f59d809cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59de84f700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 0x7f59d809cce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59d80a3350 tx=0x7f59d00099b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d000dd90 con 0x7f59d8137b60 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: dump: 2023-09-30T00:44:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59d000f040 con 0x7f59d8137b60 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 5 monclient: authenticate success, global_id 4365 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59d0015650 con 0x7f59d8137b60 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59ce7fc700 10 monclient: dump: 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59dc84b700 4 set_mon_vals no callback set 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 10 monclient: _renew_subs 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59d81362c0 con 0x7f59d8137b60 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 10 monclient: _renew_subs 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:55.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.060+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59d8133b10 con 0x7f59d8137b60 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59e02b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59e02b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59e02b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59d8072b00 con 0x7f59d8137b60 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f59d0015a40 con 0x7f59d8137b60 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59ce7fc700 1 --2- 172.21.15.6:0/1561518233 >> v2:172.21.15.22:6812/142603 conn(0x7f59bc0357a0 0x7f59bc037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.061+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f59d0048040 con 0x7f59d8137b60 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.062+0000 7f59de04e700 1 --2- 172.21.15.6:0/1561518233 >> v2:172.21.15.22:6812/142603 conn(0x7f59bc0357a0 0x7f59bc037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:55.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.062+0000 7f59de04e700 10 monclient: get_auth_request con 0x7f59bc0357a0 auth_method 0 2023-09-30T00:44:55.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.064+0000 7f59de04e700 1 --2- 172.21.15.6:0/1561518233 >> v2:172.21.15.22:6812/142603 conn(0x7f59bc0357a0 0x7f59bc037c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59d40052e0 tx=0x7f59d4004a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:55.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.066+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f59d001d360 con 0x7f59d8137b60 2023-09-30T00:44:55.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.066+0000 7f59ce7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:55.081 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.066+0000 7f59ce7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59e02b2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59e02b2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59e02b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f59d81332c0 con 0x7f59d8137b60 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59ce7fc700 1 -- 172.21.15.6:0/1561518233 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f59d0013af0 con 0x7f59d8137b60 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59ce7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.316+0000 7f59ce7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:55.318 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 >> v2:172.21.15.22:6812/142603 conn(0x7f59bc0357a0 msgr2=0x7f59bc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 --2- 172.21.15.6:0/1561518233 >> v2:172.21.15.22:6812/142603 conn(0x7f59bc0357a0 0x7f59bc037c50 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f59d40052e0 tx=0x7f59d4004a80 comp rx=0 tx=0).stop 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 10 monclient: shutdown 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 msgr2=0x7f59d809cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 0x7f59d809cce0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f59d80a3350 tx=0x7f59d00099b0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 shutdown_connections 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 --2- 172.21.15.6:0/1561518233 >> v2:172.21.15.22:6812/142603 conn(0x7f59bc0357a0 0x7f59bc037c50 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f59d8137b60 0x7f59d809cce0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f59d8098670 0x7f59d812f450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 --2- 172.21.15.6:0/1561518233 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f59d8132d00 0x7f59d809c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.318+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 >> 172.21.15.6:0/1561518233 conn(0x7f59d80a5480 msgr2=0x7f59d8131b60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.319+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 shutdown_connections 2023-09-30T00:44:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:55.319+0000 7f59e02b2700 1 -- 172.21.15.6:0/1561518233 wait complete. 2023-09-30T00:44:58.333 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:44:58.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.471+0000 7fbd65e8a700 10 monclient: get_monmap_and_config 2023-09-30T00:44:58.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.471+0000 7fbd65e8a700 10 monclient: build_initial_monmap 2023-09-30T00:44:58.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.471+0000 7fbd65e8a700 10 monclient: monmap: 2023-09-30T00:44:58.472 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:58.472 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:44:58.472659+0000 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:44:58.472659+0000 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:44:58.473 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 1 Processor -- start 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 1 -- start start 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient: init 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient(hunting): picked mon.noname-b con 0x7fbd60098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fbd60098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:44:58.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd60098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd600a1000 con 0x7fbd60098e90 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd6012fc00 con 0x7fbd60098a40 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd600a0a50 con 0x7fbd60098670 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.472+0000 7fbd65e8a700 10 monclient(hunting): authenticate will time out at 1294.770957s 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5ffff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd60098e90 0x7fbd6012f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33744/0 (socket says 172.21.15.6:33744) 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:53342/0 (socket says 172.21.15.6:53342) 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5ffff700 1 -- 172.21.15.6:0/605420399 learned_addr learned my addr 172.21.15.6:0/605420399 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:44:58.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): get_auth_request con 0x7fbd60098670 auth_method 0 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_done global_id 4364 payload 274 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/605420399 >> v1:172.21.15.22:6789/0 conn(0x7fbd60098a40 legacy=0x7fbd6012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/605420399 >> v1:172.21.15.6:6789/0 conn(0x7fbd60098e90 legacy=0x7fbd6012f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 1 monclient: found mon.noname-b 2023-09-30T00:44:58.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.473+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/605420399 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd600a0730 con 0x7fbd60098670 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/605420399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 msgr2=0x7fbd60132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5f7fe700 1 --2- 172.21.15.6:0/605420399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60132d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009910 tx=0x7fbd500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1d541f9086787101 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 1 -- 172.21.15.6:0/605420399 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000b9f0 con 0x7fbd60098670 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: dump: 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:58.481 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd65e8a700 5 monclient: authenticate success, global_id 4364 2023-09-30T00:44:58.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd65e8a700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 1 -- 172.21.15.6:0/605420399 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd5000bb50 con 0x7fbd60098670 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd65e8a700 10 monclient: get_monmap_and_config success 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd65e8a700 4 set_mon_vals no callback set 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 1 -- 172.21.15.6:0/605420399 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd50011420 con 0x7fbd60098670 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.475+0000 7fbd5e7fc700 10 monclient: dump: 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.483 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 10 monclient: shutdown 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 -- 172.21.15.6:0/605420399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 msgr2=0x7fbd60132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/605420399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60132d00 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbd50009910 tx=0x7fbd500091a0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 -- 172.21.15.6:0/605420399 shutdown_connections 2023-09-30T00:44:58.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/605420399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60132d00 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 -- 172.21.15.6:0/605420399 >> 172.21.15.6:0/605420399 conn(0x7fbd600a5480 msgr2=0x7fbd600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 -- 172.21.15.6:0/605420399 shutdown_connections 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 1 -- 172.21.15.6:0/605420399 wait complete. 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 10 monclient: build_initial_monmap 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.476+0000 7fbd65e8a700 10 monclient: monmap: 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:58.485 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:44:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:44:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:44:58.486 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 1 Processor -- start 2023-09-30T00:44:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 1 -- start start 2023-09-30T00:44:58.486 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 10 monclient: init 2023-09-30T00:44:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 10 monclient: _reopen_session rank -1 2023-09-30T00:44:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:44:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 10 monclient(hunting): picked mon.noname-c con 0x7fbd60098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:44:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 0x7fbd6012c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.477+0000 7fbd65e8a700 10 monclient(hunting): picked mon.noname-a con 0x7fbd60098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd60098e90 0x7fbd6012c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 10 monclient(hunting): picked mon.noname-b con 0x7fbd60098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd60142740 con 0x7fbd60098a40 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6009e6e0 con 0x7fbd60098e90 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6012ed70 con 0x7fbd60098670 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 10 monclient(hunting): _renew_subs 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd65e8a700 10 monclient(hunting): authenticate will time out at 1294.776035s 2023-09-30T00:44:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 0x7fbd6012c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:53352/0 (socket says 172.21.15.6:53352) 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/4019689884 learned_addr learned my addr 172.21.15.6:0/4019689884 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 0x7fbd6012c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47894/0 (socket says 172.21.15.6:47894) 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): get_auth_request con 0x7fbd60098670 auth_method 0 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): get_auth_request con 0x7fbd60098a40 auth_method 0 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd60098e90 0x7fbd6012c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): get_auth_request con 0x7fbd60098e90 auth_method 0 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:44:58.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient(hunting): handle_auth_done global_id 4367 payload 824 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/4019689884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd60098e90 msgr2=0x7fbd6012c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd60098e90 0x7fbd6012c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 msgr2=0x7fbd6012c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 0x7fbd6012c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 monclient: found mon.noname-c 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:44:58.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/4019689884 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd50009420 con 0x7fbd60098670 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.478+0000 7fbd5effd700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 0x7fbd6012c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd5f7fe700 1 -- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 msgr2=0x7fbd60141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd5f7fe700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60141840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd6006a6b0 tx=0x7fbd50016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5000bed0 con 0x7fbd60098670 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd477fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd477fe700 10 monclient: dump: 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.495 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.480+0000 7fbd477fe700 10 monclient: _finish_auth 0 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd50020070 con 0x7fbd60098670 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 5 monclient: authenticate success, global_id 4367 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 10 monclient: _renew_subs 2023-09-30T00:44:58.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd6011c350 con 0x7fbd60098670 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd5001b6a0 con 0x7fbd60098670 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 10 monclient: dump: 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:44:58.497 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 10 monclient: _renew_subs 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd6009f220 con 0x7fbd60098670 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd5dffb700 4 set_mon_vals no callback set 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.481+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbd50022030 con 0x7fbd60098670 2023-09-30T00:44:58.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd65e8a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd65e8a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd65e8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd477fe700 1 --2- 172.21.15.6:0/4019689884 >> v2:172.21.15.22:6812/142603 conn(0x7fbd4c0357a0 0x7fbd4c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6009f220 con 0x7fbd60098670 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbd50049710 con 0x7fbd60098670 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd5effd700 1 --2- 172.21.15.6:0/4019689884 >> v2:172.21.15.22:6812/142603 conn(0x7fbd4c0357a0 0x7fbd4c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.482+0000 7fbd5effd700 10 monclient: get_auth_request con 0x7fbd4c0357a0 auth_method 0 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.485+0000 7fbd5effd700 1 --2- 172.21.15.6:0/4019689884 >> v2:172.21.15.22:6812/142603 conn(0x7fbd4c0357a0 0x7fbd4c037c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbd6012fbd0 tx=0x7fbd54007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.487+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd5001a9c0 con 0x7fbd60098670 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.487+0000 7fbd477fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:44:58.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.487+0000 7fbd477fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:44:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd65e8a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:44:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd65e8a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd65e8a700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:44:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd6009f220 con 0x7fbd60098670 2023-09-30T00:44:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd477fe700 1 -- 172.21.15.6:0/4019689884 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbd5001a3d0 con 0x7fbd60098670 2023-09-30T00:44:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd477fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:44:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.732+0000 7fbd477fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:44:58.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:44:58.734 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 >> v2:172.21.15.22:6812/142603 conn(0x7fbd4c0357a0 msgr2=0x7fbd4c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/4019689884 >> v2:172.21.15.22:6812/142603 conn(0x7fbd4c0357a0 0x7fbd4c037c50 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fbd6012fbd0 tx=0x7fbd54007ea0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 10 monclient: shutdown 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 msgr2=0x7fbd60141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60141840 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fbd6006a6b0 tx=0x7fbd50016000 comp rx=0 tx=0).stop 2023-09-30T00:44:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 shutdown_connections 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/4019689884 >> v2:172.21.15.22:6812/142603 conn(0x7fbd4c0357a0 0x7fbd4c037c50 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd60098e90 0x7fbd6012c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd60098670 0x7fbd60141840 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 --2- 172.21.15.6:0/4019689884 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd60098a40 0x7fbd6012c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 >> 172.21.15.6:0/4019689884 conn(0x7fbd600a5480 msgr2=0x7fbd6012e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 shutdown_connections 2023-09-30T00:44:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:44:58.735+0000 7fbd65e8a700 1 -- 172.21.15.6:0/4019689884 wait complete. 2023-09-30T00:45:01.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:01.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.886+0000 7ff592210700 10 monclient: get_monmap_and_config 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.886+0000 7ff592210700 10 monclient: build_initial_monmap 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.887+0000 7ff592210700 10 monclient: monmap: 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:01.888202+0000 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:01.888202+0000 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:01.888 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:01.889 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:01.889 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:01.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.887+0000 7ff592210700 1 Processor -- start 2023-09-30T00:45:01.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.887+0000 7ff592210700 1 -- start start 2023-09-30T00:45:01.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.887+0000 7ff592210700 10 monclient: init 2023-09-30T00:45:01.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7ff58c098670 0x7ff58c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): picked mon.noname-c con 0x7ff58c098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 0x7ff58c12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): picked mon.noname-b con 0x7ff58c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff58c098e90 0x7ff58c12f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): picked mon.noname-a con 0x7ff58c098e90 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c0a0a50 con 0x7ff58c098e90 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:01.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff58c0a1000 con 0x7ff58c098670 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0a0730 con 0x7ff58c098a40 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff592210700 10 monclient(hunting): authenticate will time out at 1298.186358s 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff58c098e90 0x7ff58c12f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 0x7ff58c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff58c098e90 0x7ff58c12f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47898/0 (socket says 172.21.15.6:47898) 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 0x7ff58c12ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:53364/0 (socket says 172.21.15.6:53364) 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 -- 172.21.15.6:0/3744472915 learned_addr learned my addr 172.21.15.6:0/3744472915 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): get_auth_request con 0x7ff58c098e90 auth_method 0 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:01.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): get_auth_request con 0x7ff58c098a40 auth_method 0 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:01.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58b7fe700 1 --2- 172.21.15.6:0/3744472915 >> v2:172.21.15.22:3300/0 conn(0x7ff58c098670 0x7ff58c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58b7fe700 10 monclient(hunting): get_auth_request con 0x7ff58c098670 auth_method 0 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient(hunting): handle_auth_done global_id 4371 payload 274 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 -- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 msgr2=0x7ff58c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 --2- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 0x7ff58c12ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 -- 172.21.15.6:0/3744472915 >> v2:172.21.15.22:3300/0 conn(0x7ff58c098670 msgr2=0x7ff58c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 --2- 172.21.15.6:0/3744472915 >> v2:172.21.15.22:3300/0 conn(0x7ff58c098670 0x7ff58c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 monclient: found mon.noname-a 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58bfff700 1 -- 172.21.15.6:0/3744472915 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff58c068df0 con 0x7ff58c098e90 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58b7fe700 1 --2- 172.21.15.6:0/3744472915 >> v2:172.21.15.22:3300/0 conn(0x7ff58c098670 0x7ff58c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.888+0000 7ff58affd700 1 --2- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 0x7ff58c12ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58bfff700 1 -- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098e90 msgr2=0x7ff58c12f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58bfff700 1 --2- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098e90 0x7ff58c12f320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff58c09fe60 tx=0x7ff584009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ca4734c4e8bd56c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 1 -- 172.21.15.6:0/3744472915 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff58400e070 con 0x7ff58c098e90 2023-09-30T00:45:01.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: dump: 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:01.898 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 1 -- 172.21.15.6:0/3744472915 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff58400bc30 con 0x7ff58c098e90 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 5 monclient: authenticate success, global_id 4371 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 1 -- 172.21.15.6:0/3744472915 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5840092e0 con 0x7ff58c098e90 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 4 set_mon_vals no callback set 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:01.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff58a7fc700 10 monclient: dump: 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 10 monclient: shutdown 2023-09-30T00:45:01.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 -- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098e90 msgr2=0x7ff58c12f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 --2- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098e90 0x7ff58c12f320 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff58c09fe60 tx=0x7ff584009840 comp rx=0 tx=0).stop 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 -- 172.21.15.6:0/3744472915 shutdown_connections 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 --2- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098e90 0x7ff58c12f320 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 --2- 172.21.15.6:0/3744472915 >> v2:172.21.15.22:3300/0 conn(0x7ff58c098670 0x7ff58c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 --2- 172.21.15.6:0/3744472915 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098a40 0x7ff58c12ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.891+0000 7ff592210700 1 -- 172.21.15.6:0/3744472915 >> 172.21.15.6:0/3744472915 conn(0x7ff58c0a5480 msgr2=0x7ff58c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.892+0000 7ff592210700 1 -- 172.21.15.6:0/3744472915 shutdown_connections 2023-09-30T00:45:01.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.892+0000 7ff592210700 1 -- 172.21.15.6:0/3744472915 wait complete. 2023-09-30T00:45:01.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.892+0000 7ff592210700 10 monclient: build_initial_monmap 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.892+0000 7ff592210700 10 monclient: monmap: 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:01.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 Processor -- start 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 -- start start 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient: init 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 0x7ff58c12f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): picked mon.noname-a con 0x7ff58c098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff58c098a40 0x7ff58c141740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): picked mon.noname-b con 0x7ff58c098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098e90 0x7ff58c141c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): picked mon.noname-c con 0x7ff58c098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:01.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c0a37e0 con 0x7ff58c098670 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c12d2c0 con 0x7ff58c098a40 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff58c12d540 con 0x7ff58c098e90 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff592210700 10 monclient(hunting): authenticate will time out at 1298.191679s 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 0x7ff58c12f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 0x7ff58c12f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47914/0 (socket says 172.21.15.6:47914) 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 1 -- 172.21.15.6:0/2790580089 learned_addr learned my addr 172.21.15.6:0/2790580089 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): get_auth_request con 0x7ff58c098670 auth_method 0 2023-09-30T00:45:01.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098e90 0x7ff58c141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): get_auth_request con 0x7ff58c098e90 auth_method 0 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.893+0000 7ff58affd700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff58c098a40 0x7ff58c141740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58affd700 10 monclient(hunting): get_auth_request con 0x7ff58c098a40 auth_method 0 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 10 monclient(hunting): handle_auth_done global_id 4374 payload 824 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 1 -- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098e90 msgr2=0x7ff58c141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098e90 0x7ff58c141c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 1 -- 172.21.15.6:0/2790580089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff58c098a40 msgr2=0x7ff58c141740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff58c098a40 0x7ff58c141740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:45:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 1 monclient: found mon.noname-a 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58b7fe700 1 -- 172.21.15.6:0/2790580089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff580009420 con 0x7ff58c098670 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.894+0000 7ff58affd700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff58c098a40 0x7ff58c141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff58b7fe700 1 -- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 msgr2=0x7ff58c12f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff58b7fe700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 0x7ff58c12f030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00e8d0 tx=0x7ff57c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c013620 con 0x7ff58c098670 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: dump: 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:01.911 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: _finish_auth 0 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff57c013780 con 0x7ff58c098670 2023-09-30T00:45:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 5 monclient: authenticate success, global_id 4374 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 10 monclient: _renew_subs 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff584009ae0 con 0x7ff58c098670 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff57c013b20 con 0x7ff58c098670 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff5737fe700 10 monclient: dump: 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff589ffb700 4 set_mon_vals no callback set 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 10 monclient: _renew_subs 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.896+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff58c09eec0 con 0x7ff58c098670 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff57c00c3a0 con 0x7ff58c098670 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff592210700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff592210700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff592210700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff58c072b00 con 0x7ff58c098670 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff5737fe700 1 --2- 172.21.15.6:0/2790580089 >> v2:172.21.15.22:6812/142603 conn(0x7ff5780357f0 0x7ff578037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.897+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff57c0477c0 con 0x7ff58c098670 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.898+0000 7ff58affd700 1 --2- 172.21.15.6:0/2790580089 >> v2:172.21.15.22:6812/142603 conn(0x7ff5780357f0 0x7ff578037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.898+0000 7ff58affd700 10 monclient: get_auth_request con 0x7ff5780357f0 auth_method 0 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.900+0000 7ff58affd700 1 --2- 172.21.15.6:0/2790580089 >> v2:172.21.15.22:6812/142603 conn(0x7ff5780357f0 0x7ff578037ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff5800093f0 tx=0x7ff580008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.902+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff57c022af0 con 0x7ff58c098670 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.902+0000 7ff5737fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:01.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:01.902+0000 7ff5737fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.148+0000 7ff592210700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.148+0000 7ff592210700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.148+0000 7ff592210700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.148+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff58c098250 con 0x7ff58c098670 2023-09-30T00:45:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.149+0000 7ff5737fe700 1 -- 172.21.15.6:0/2790580089 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff57c020070 con 0x7ff58c098670 2023-09-30T00:45:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.149+0000 7ff5737fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.149+0000 7ff5737fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:02.150 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:02.150 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 >> v2:172.21.15.22:6812/142603 conn(0x7ff5780357f0 msgr2=0x7ff578037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 --2- 172.21.15.6:0/2790580089 >> v2:172.21.15.22:6812/142603 conn(0x7ff5780357f0 0x7ff578037ca0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff5800093f0 tx=0x7ff580008040 comp rx=0 tx=0).stop 2023-09-30T00:45:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 10 monclient: shutdown 2023-09-30T00:45:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 msgr2=0x7ff58c12f030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 0x7ff58c12f030 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff57c00e8d0 tx=0x7ff57c008040 comp rx=0 tx=0).stop 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 shutdown_connections 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 --2- 172.21.15.6:0/2790580089 >> v2:172.21.15.22:6812/142603 conn(0x7ff5780357f0 0x7ff578037ca0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff58c098e90 0x7ff58c141c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff58c098670 0x7ff58c12f030 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 --2- 172.21.15.6:0/2790580089 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff58c098a40 0x7ff58c141740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 >> 172.21.15.6:0/2790580089 conn(0x7ff58c0a5480 msgr2=0x7ff58c069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 shutdown_connections 2023-09-30T00:45:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:02.151+0000 7ff592210700 1 -- 172.21.15.6:0/2790580089 wait complete. 2023-09-30T00:45:05.166 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.304+0000 7f6a3fda8700 10 monclient: get_monmap_and_config 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.304+0000 7f6a3fda8700 10 monclient: build_initial_monmap 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.304+0000 7f6a3fda8700 10 monclient: monmap: 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:05.305286+0000 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:05.305286+0000 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:05.305 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:05.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.304+0000 7f6a3fda8700 1 Processor -- start 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.304+0000 7f6a3fda8700 1 -- start start 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.304+0000 7f6a3fda8700 10 monclient: init 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a38098670 0x7f6a38132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): picked mon.noname-a con 0x7f6a38098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6a38098a40 0x7f6a3812ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:05.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): picked mon.noname-c con 0x7f6a38098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6a38098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a380a1000 con 0x7f6a38098e90 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380a0a50 con 0x7f6a38098670 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a380a0730 con 0x7f6a38098a40 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3fda8700 10 monclient(hunting): authenticate will time out at 1301.603544s 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a38098670 0x7f6a38132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6a38098670 0x7f6a38132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:47920/0 (socket says 172.21.15.6:47920) 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 1 -- 172.21.15.6:0/3100538578 learned_addr learned my addr 172.21.15.6:0/3100538578 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:05.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): get_auth_request con 0x7f6a38098670 auth_method 0 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3d343700 1 --2- 172.21.15.6:0/3100538578 >> v2:172.21.15.22:3300/0 conn(0x7f6a38098a40 0x7f6a3812ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3d343700 10 monclient(hunting): get_auth_request con 0x7f6a38098a40 auth_method 0 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3d343700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3d343700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3d343700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:05.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 10 monclient(hunting): handle_auth_done global_id 4377 payload 274 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.305+0000 7f6a3db44700 1 -- 172.21.15.6:0/3100538578 >> v2:172.21.15.22:3300/0 conn(0x7f6a38098a40 msgr2=0x7f6a3812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 1 --2- 172.21.15.6:0/3100538578 >> v2:172.21.15.22:3300/0 conn(0x7f6a38098a40 0x7f6a3812ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 1 -- 172.21.15.6:0/3100538578 >> v1:172.21.15.22:6789/0 conn(0x7f6a38098e90 legacy=0x7f6a3812f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 10 monclient: _finish_hunting 0 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 1 monclient: found mon.noname-a 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.306+0000 7f6a3db44700 1 -- 172.21.15.6:0/3100538578 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a38068df0 con 0x7f6a38098670 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3db44700 1 -- 172.21.15.6:0/3100538578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098670 msgr2=0x7f6a38132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3db44700 1 --2- 172.21.15.6:0/3100538578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098670 0x7f6a38132d00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6a38063790 tx=0x7f6a34009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a2c2e100c986d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 1 -- 172.21.15.6:0/3100538578 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a3400ba30 con 0x7f6a38098670 2023-09-30T00:45:05.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: dump: 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:05.314 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: _finish_auth 0 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 1 -- 172.21.15.6:0/3100538578 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a3400bb90 con 0x7f6a38098670 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3fda8700 5 monclient: authenticate success, global_id 4377 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3fda8700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 1 -- 172.21.15.6:0/3100538578 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a34011420 con 0x7f6a38098670 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3fda8700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3fda8700 4 set_mon_vals no callback set 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.308+0000 7f6a3cb42700 10 monclient: dump: 2023-09-30T00:45:05.315 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 10 monclient: shutdown 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:05.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 -- 172.21.15.6:0/3100538578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098670 msgr2=0x7f6a38132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/3100538578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098670 0x7f6a38132d00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6a38063790 tx=0x7f6a34009250 comp rx=0 tx=0).stop 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 -- 172.21.15.6:0/3100538578 shutdown_connections 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/3100538578 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098670 0x7f6a38132d00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/3100538578 >> v2:172.21.15.22:3300/0 conn(0x7f6a38098a40 0x7f6a3812ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 -- 172.21.15.6:0/3100538578 >> 172.21.15.6:0/3100538578 conn(0x7f6a380a5480 msgr2=0x7f6a380a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 -- 172.21.15.6:0/3100538578 shutdown_connections 2023-09-30T00:45:05.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.309+0000 7f6a3fda8700 1 -- 172.21.15.6:0/3100538578 wait complete. 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.310+0000 7f6a3fda8700 10 monclient: build_initial_monmap 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.310+0000 7f6a3fda8700 10 monclient: monmap: 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:05.322 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.310+0000 7f6a3fda8700 1 Processor -- start 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.310+0000 7f6a3fda8700 1 -- start start 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.310+0000 7f6a3fda8700 10 monclient: init 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 0x7f6a3809f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): picked mon.noname-c con 0x7f6a38098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a38098a40 0x7f6a3812ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): picked mon.noname-b con 0x7f6a38098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:05.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 0x7f6a3812f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): picked mon.noname-a con 0x7f6a38098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3809c7a0 con 0x7f6a38098e90 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3809d410 con 0x7f6a38098a40 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a3809f6c0 con 0x7f6a38098670 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3fda8700 10 monclient(hunting): authenticate will time out at 1301.609416s 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 0x7f6a3809f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 0x7f6a3809f180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:53386/0 (socket says 172.21.15.6:53386) 2023-09-30T00:45:05.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 1 -- 172.21.15.6:0/2346370375 learned_addr learned my addr 172.21.15.6:0/2346370375 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): get_auth_request con 0x7f6a38098670 auth_method 0 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 0x7f6a3812f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): get_auth_request con 0x7f6a38098e90 auth_method 0 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:05.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3d343700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a38098a40 0x7f6a3812ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3d343700 10 monclient(hunting): get_auth_request con 0x7f6a38098a40 auth_method 0 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3d343700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3d343700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3d343700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient(hunting): handle_auth_done global_id 4380 payload 824 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 -- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 msgr2=0x7f6a3809f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 0x7f6a3809f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 -- 172.21.15.6:0/2346370375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a38098a40 msgr2=0x7f6a3812ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a38098a40 0x7f6a3812ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient: _finish_hunting 0 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 monclient: found mon.noname-a 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3e345700 1 -- 172.21.15.6:0/2346370375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a340094f0 con 0x7f6a38098e90 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3db44700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 0x7f6a3809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.311+0000 7f6a3d343700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a38098a40 0x7f6a3812ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3e345700 1 -- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 msgr2=0x7f6a3812f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3e345700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 0x7f6a3812f010 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000c810 tx=0x7f6a30002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30012840 con 0x7f6a38098e90 2023-09-30T00:45:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: dump: 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:05.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: _finish_auth 0 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 5 monclient: authenticate success, global_id 4380 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a300129a0 con 0x7f6a38098e90 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 10 monclient: _renew_subs 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a3809e200 con 0x7f6a38098e90 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6a30012d60 con 0x7f6a38098e90 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2dffb700 10 monclient: dump: 2023-09-30T00:45:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a2ffff700 4 set_mon_vals no callback set 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 10 monclient: _renew_subs 2023-09-30T00:45:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.314+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a3812f550 con 0x7f6a38098e90 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6a30011590 con 0x7f6a38098e90 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a3fda8700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a3fda8700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a3fda8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a38098110 con 0x7f6a38098e90 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a2dffb700 1 --2- 172.21.15.6:0/2346370375 >> v2:172.21.15.22:6812/142603 conn(0x7f6a1c0357a0 0x7f6a1c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6a300475f0 con 0x7f6a38098e90 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a3db44700 1 --2- 172.21.15.6:0/2346370375 >> v2:172.21.15.22:6812/142603 conn(0x7f6a1c0357a0 0x7f6a1c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.315+0000 7f6a3db44700 10 monclient: get_auth_request con 0x7f6a1c0357a0 auth_method 0 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.318+0000 7f6a3db44700 1 --2- 172.21.15.6:0/2346370375 >> v2:172.21.15.22:6812/142603 conn(0x7f6a1c0357a0 0x7f6a1c037c50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a340030a0 tx=0x7f6a34016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.320+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6a300184a0 con 0x7f6a38098e90 2023-09-30T00:45:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.320+0000 7f6a2dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.320+0000 7f6a2dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a3fda8700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a3fda8700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a3fda8700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6a38072b00 con 0x7f6a38098e90 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a2dffb700 1 -- 172.21.15.6:0/2346370375 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6a30016020 con 0x7f6a38098e90 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a2dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.566+0000 7f6a2dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:05.568 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:05.568 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.568+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 >> v2:172.21.15.22:6812/142603 conn(0x7f6a1c0357a0 msgr2=0x7f6a1c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.568+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/2346370375 >> v2:172.21.15.22:6812/142603 conn(0x7f6a1c0357a0 0x7f6a1c037c50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6a340030a0 tx=0x7f6a34016000 comp rx=0 tx=0).stop 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.568+0000 7f6a3fda8700 10 monclient: shutdown 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.568+0000 7f6a3fda8700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.568+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 msgr2=0x7f6a3812f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.568+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 0x7f6a3812f010 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6a3000c810 tx=0x7f6a30002620 comp rx=0 tx=0).stop 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 shutdown_connections 2023-09-30T00:45:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/2346370375 >> v2:172.21.15.22:6812/142603 conn(0x7f6a1c0357a0 0x7f6a1c037c50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6a38098e90 0x7f6a3812f010 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6a38098670 0x7f6a3809f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 --2- 172.21.15.6:0/2346370375 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6a38098a40 0x7f6a3812ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 >> 172.21.15.6:0/2346370375 conn(0x7f6a380a5480 msgr2=0x7f6a38133370 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 shutdown_connections 2023-09-30T00:45:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:05.569+0000 7f6a3fda8700 1 -- 172.21.15.6:0/2346370375 wait complete. 2023-09-30T00:45:08.584 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:08.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.722+0000 7f2b28432700 10 monclient: get_monmap_and_config 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.722+0000 7f2b28432700 10 monclient: build_initial_monmap 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.722+0000 7f2b28432700 10 monclient: monmap: 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:08.723268+0000 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:08.723268+0000 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:08.723 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:08.724 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:08.724 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:08.724 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:08.724 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.722+0000 7f2b28432700 1 Processor -- start 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.722+0000 7f2b28432700 1 -- start start 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.722+0000 7f2b28432700 10 monclient: init 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient(hunting): picked mon.noname-b con 0x7f2b20098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:08.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2b20098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2b20098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b200a1000 con 0x7f2b20098e90 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b2012fc00 con 0x7f2b20098a40 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b200a0a50 con 0x7f2b20098670 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b28432700 10 monclient(hunting): authenticate will time out at 1305.021572s 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51558/0 (socket says 172.21.15.6:51558) 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 1 -- 172.21.15.6:0/1887134134 learned_addr learned my addr 172.21.15.6:0/1887134134 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): get_auth_request con 0x7f2b20098670 auth_method 0 2023-09-30T00:45:08.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 10 monclient(hunting): handle_auth_done global_id 4379 payload 274 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.723+0000 7f2b261ce700 1 -- 172.21.15.6:0/1887134134 >> v1:172.21.15.22:6789/0 conn(0x7f2b20098a40 legacy=0x7f2b2012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.724+0000 7f2b261ce700 1 -- 172.21.15.6:0/1887134134 >> v1:172.21.15.6:6789/0 conn(0x7f2b20098e90 legacy=0x7f2b2012f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.724+0000 7f2b261ce700 10 monclient: _finish_hunting 0 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.724+0000 7f2b261ce700 1 monclient: found mon.noname-b 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.724+0000 7f2b261ce700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:08.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.724+0000 7f2b261ce700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:08.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.724+0000 7f2b261ce700 1 -- 172.21.15.6:0/1887134134 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b200a0730 con 0x7f2b20098670 2023-09-30T00:45:08.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b261ce700 1 -- 172.21.15.6:0/1887134134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 msgr2=0x7f2b20132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b261ce700 1 --2- 172.21.15.6:0/1887134134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20132d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0099e0 tx=0x7f2b1c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbc5e0718910c9b3 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 1 -- 172.21.15.6:0/1887134134 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c00bad0 con 0x7f2b20098670 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: dump: 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:08.734 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: _finish_auth 0 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 1 -- 172.21.15.6:0/1887134134 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b1c00bc30 con 0x7f2b20098670 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b28432700 5 monclient: authenticate success, global_id 4379 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b28432700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b28432700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:08.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 1 -- 172.21.15.6:0/1887134134 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c011420 con 0x7f2b20098670 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b28432700 4 set_mon_vals no callback set 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b251cc700 10 monclient: dump: 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:08.736 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b28432700 10 monclient: shutdown 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.726+0000 7f2b28432700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 -- 172.21.15.6:0/1887134134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 msgr2=0x7f2b20132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 --2- 172.21.15.6:0/1887134134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20132d00 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c0099e0 tx=0x7f2b1c009250 comp rx=0 tx=0).stop 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 -- 172.21.15.6:0/1887134134 shutdown_connections 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 --2- 172.21.15.6:0/1887134134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20132d00 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 -- 172.21.15.6:0/1887134134 >> 172.21.15.6:0/1887134134 conn(0x7f2b200a5480 msgr2=0x7f2b200a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 -- 172.21.15.6:0/1887134134 shutdown_connections 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 1 -- 172.21.15.6:0/1887134134 wait complete. 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 10 monclient: build_initial_monmap 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.727+0000 7f2b28432700 10 monclient: monmap: 2023-09-30T00:45:08.737 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 Processor -- start 2023-09-30T00:45:08.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 -- start start 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient: init 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): picked mon.noname-c con 0x7f2b20098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 0x7f2b2012c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): picked mon.noname-a con 0x7f2b20098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2b20098e90 0x7f2b2012c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): picked mon.noname-b con 0x7f2b20098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b20142740 con 0x7f2b20098a40 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:08.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2009e6e0 con 0x7f2b20098e90 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b2012ed70 con 0x7f2b20098670 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b28432700 10 monclient(hunting): authenticate will time out at 1305.026598s 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b261ce700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b261ce700 1 -- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 msgr2=0x7f2b20141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b261ce700 1 -- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 msgr2=0x7f2b20141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b261ce700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b261ce700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20141840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 0x7f2b2012c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b269cf700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2b20098e90 0x7f2b2012c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:08.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 0x7f2b2012c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38770/0 (socket says 172.21.15.6:38770) 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 1 -- 172.21.15.6:0/2871071297 learned_addr learned my addr 172.21.15.6:0/2871071297 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): get_auth_request con 0x7f2b20098a40 auth_method 0 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b269cf700 10 monclient(hunting): get_auth_request con 0x7f2b20098e90 auth_method 0 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b269cf700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b269cf700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b269cf700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b259cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:08.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.728+0000 7f2b269cf700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b269cf700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b269cf700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 10 monclient(hunting): handle_auth_done global_id 4383 payload 824 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 1 -- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 msgr2=0x7f2b20141840 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20141840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 1 -- 172.21.15.6:0/2871071297 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2b20098e90 msgr2=0x7f2b2012c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2b20098e90 0x7f2b2012c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 10 monclient: _finish_hunting 0 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 1 monclient: found mon.noname-a 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:08.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.729+0000 7f2b259cd700 1 -- 172.21.15.6:0/2871071297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b1c0094f0 con 0x7f2b20098a40 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b259cd700 1 -- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 msgr2=0x7f2b2012c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b259cd700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 0x7f2b2012c200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c5d0 tx=0x7f2b10007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b100117a0 con 0x7f2b20098a40 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: dump: 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.743 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: _finish_auth 0 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 5 monclient: authenticate success, global_id 4383 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 10 monclient: _renew_subs 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b10011900 con 0x7f2b20098a40 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b2011c350 con 0x7f2b20098a40 2023-09-30T00:45:08.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2b10011cc0 con 0x7f2b20098a40 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b167fc700 10 monclient: dump: 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:08.745 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b249cb700 4 set_mon_vals no callback set 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 10 monclient: _renew_subs 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.731+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b20141d80 con 0x7f2b20098a40 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2b10012630 con 0x7f2b20098a40 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b167fc700 1 --2- 172.21.15.6:0/2871071297 >> v2:172.21.15.22:6812/142603 conn(0x7f2b040357a0 0x7f2b04037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2b10047690 con 0x7f2b20098a40 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b28432700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b28432700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b28432700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:08.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b20141fa0 con 0x7f2b20098a40 2023-09-30T00:45:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b261ce700 1 --2- 172.21.15.6:0/2871071297 >> v2:172.21.15.22:6812/142603 conn(0x7f2b040357a0 0x7f2b04037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.732+0000 7f2b261ce700 10 monclient: get_auth_request con 0x7f2b040357a0 auth_method 0 2023-09-30T00:45:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.735+0000 7f2b261ce700 1 --2- 172.21.15.6:0/2871071297 >> v2:172.21.15.22:6812/142603 conn(0x7f2b040357a0 0x7f2b04037c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00bfd0 tx=0x7f2b1c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.737+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2b10011e60 con 0x7f2b20098a40 2023-09-30T00:45:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.737+0000 7f2b167fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:08.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.737+0000 7f2b167fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:08.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.981+0000 7f2b28432700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.981+0000 7f2b28432700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.981+0000 7f2b28432700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.981+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2b20072b00 con 0x7f2b20098a40 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.982+0000 7f2b167fc700 1 -- 172.21.15.6:0/2871071297 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2b100102e0 con 0x7f2b20098a40 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.982+0000 7f2b167fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.982+0000 7f2b167fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:08.983 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:08.984 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 >> v2:172.21.15.22:6812/142603 conn(0x7f2b040357a0 msgr2=0x7f2b04037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 --2- 172.21.15.6:0/2871071297 >> v2:172.21.15.22:6812/142603 conn(0x7f2b040357a0 0x7f2b04037c50 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00bfd0 tx=0x7f2b1c016000 comp rx=0 tx=0).stop 2023-09-30T00:45:08.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 10 monclient: shutdown 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 msgr2=0x7f2b2012c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 0x7f2b2012c200 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f2b1000c5d0 tx=0x7f2b10007f60 comp rx=0 tx=0).stop 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 shutdown_connections 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 --2- 172.21.15.6:0/2871071297 >> v2:172.21.15.22:6812/142603 conn(0x7f2b040357a0 0x7f2b04037c50 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2b20098e90 0x7f2b2012c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2b20098670 0x7f2b20141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 --2- 172.21.15.6:0/2871071297 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2b20098a40 0x7f2b2012c200 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 >> 172.21.15.6:0/2871071297 conn(0x7f2b200a5480 msgr2=0x7f2b2012e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 shutdown_connections 2023-09-30T00:45:08.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:08.984+0000 7f2b28432700 1 -- 172.21.15.6:0/2871071297 wait complete. 2023-09-30T00:45:11.999 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:12.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.137+0000 7f499808e700 10 monclient: get_monmap_and_config 2023-09-30T00:45:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.137+0000 7f499808e700 10 monclient: build_initial_monmap 2023-09-30T00:45:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.137+0000 7f499808e700 10 monclient: monmap: 2023-09-30T00:45:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:12.139 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:12.138271+0000 2023-09-30T00:45:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:12.138271+0000 2023-09-30T00:45:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:12.140 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:12.141 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.137+0000 7f499808e700 1 Processor -- start 2023-09-30T00:45:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.137+0000 7f499808e700 1 -- start start 2023-09-30T00:45:12.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.137+0000 7f499808e700 10 monclient: init 2023-09-30T00:45:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient: _add_conns ranks=[2,4,0,3,1] 2023-09-30T00:45:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 0x7f4990132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): picked mon.noname-b con 0x7f4990098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:12.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4990098a40 0x7f499012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): picked mon.noname-c con 0x7f4990098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4990098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49900a1000 con 0x7f4990098e90 2023-09-30T00:45:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:12.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49900a0a50 con 0x7f4990098a40 2023-09-30T00:45:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49900a0730 con 0x7f4990098670 2023-09-30T00:45:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499808e700 10 monclient(hunting): authenticate will time out at 1308.436603s 2023-09-30T00:45:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 0x7f4990132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:12.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499662b700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4990098e90 0x7f499012f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:50392/0 (socket says 172.21.15.6:50392) 2023-09-30T00:45:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f499662b700 1 -- 172.21.15.6:0/1578646950 learned_addr learned my addr 172.21.15.6:0/1578646950 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:45:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): get_auth_request con 0x7f4990098670 auth_method 0 2023-09-30T00:45:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:12.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 1 --2- 172.21.15.6:0/1578646950 >> v2:172.21.15.22:3300/0 conn(0x7f4990098a40 0x7f499012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): get_auth_request con 0x7f4990098a40 auth_method 0 2023-09-30T00:45:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:12.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995e2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.138+0000 7f4995629700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:12.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 10 monclient(hunting): handle_auth_done global_id 4382 payload 274 2023-09-30T00:45:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 1 -- 172.21.15.6:0/1578646950 >> v2:172.21.15.22:3300/0 conn(0x7f4990098a40 msgr2=0x7f499012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 1 --2- 172.21.15.6:0/1578646950 >> v2:172.21.15.22:3300/0 conn(0x7f4990098a40 0x7f499012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 1 -- 172.21.15.6:0/1578646950 >> v1:172.21.15.6:6789/0 conn(0x7f4990098e90 legacy=0x7f499012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 10 monclient: _finish_hunting 0 2023-09-30T00:45:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 1 monclient: found mon.noname-b 2023-09-30T00:45:12.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.139+0000 7f4995e2a700 1 -- 172.21.15.6:0/1578646950 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4990068df0 con 0x7f4990098670 2023-09-30T00:45:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4995e2a700 1 -- 172.21.15.6:0/1578646950 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 msgr2=0x7f4990132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4995e2a700 1 --2- 172.21.15.6:0/1578646950 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 0x7f4990132d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f498c0098e0 tx=0x7f498c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ab0d51d442bbdfed server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 1 -- 172.21.15.6:0/1578646950 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c011840 con 0x7f4990098670 2023-09-30T00:45:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: dump: 2023-09-30T00:45:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:12.168 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:12.169 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: _finish_auth 0 2023-09-30T00:45:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 1 -- 172.21.15.6:0/1578646950 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498c0119a0 con 0x7f4990098670 2023-09-30T00:45:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 5 monclient: authenticate success, global_id 4382 2023-09-30T00:45:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 1 -- 172.21.15.6:0/1578646950 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498c011d40 con 0x7f4990098670 2023-09-30T00:45:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 4 set_mon_vals no callback set 2023-09-30T00:45:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f4994e28700 10 monclient: dump: 2023-09-30T00:45:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:12.171 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:12.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:12.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:12.172 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:12.172 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:12.173 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:12.173 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:12.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 10 monclient: shutdown 2023-09-30T00:45:12.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:12.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 1 -- 172.21.15.6:0/1578646950 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 msgr2=0x7f4990132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:12.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.141+0000 7f499808e700 1 --2- 172.21.15.6:0/1578646950 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 0x7f4990132d00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f498c0098e0 tx=0x7f498c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:12.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 1 -- 172.21.15.6:0/1578646950 shutdown_connections 2023-09-30T00:45:12.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 1 --2- 172.21.15.6:0/1578646950 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098670 0x7f4990132d00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:12.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 1 --2- 172.21.15.6:0/1578646950 >> v2:172.21.15.22:3300/0 conn(0x7f4990098a40 0x7f499012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:12.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 1 -- 172.21.15.6:0/1578646950 >> 172.21.15.6:0/1578646950 conn(0x7f49900a5480 msgr2=0x7f49900a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:12.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 1 -- 172.21.15.6:0/1578646950 shutdown_connections 2023-09-30T00:45:12.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 1 -- 172.21.15.6:0/1578646950 wait complete. 2023-09-30T00:45:12.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 10 monclient: build_initial_monmap 2023-09-30T00:45:12.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.142+0000 7f499808e700 10 monclient: monmap: 2023-09-30T00:45:12.175 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:12.175 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:12.176 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:12.176 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:12.176 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:12.176 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:12.176 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:12.177 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:12.177 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:12.177 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:12.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 Processor -- start 2023-09-30T00:45:12.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 -- start start 2023-09-30T00:45:12.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient: init 2023-09-30T00:45:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:45:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 0x7f499009efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): picked mon.noname-a con 0x7f4990098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4990098a40 0x7f4990141950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:12.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): picked mon.noname-b con 0x7f4990098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:12.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:12.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): picked mon.noname-c con 0x7f4990098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:12.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:12.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f499009a9f0 con 0x7f4990098670 2023-09-30T00:45:12.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:12.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49900a1500 con 0x7f4990098a40 2023-09-30T00:45:12.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:12.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f499009e600 con 0x7f4990098e90 2023-09-30T00:45:12.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:12.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499808e700 10 monclient(hunting): authenticate will time out at 1308.441617s 2023-09-30T00:45:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 0x7f499009efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 0x7f499009efc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38776/0 (socket says 172.21.15.6:38776) 2023-09-30T00:45:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51582/0 (socket says 172.21.15.6:51582) 2023-09-30T00:45:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 1 -- 172.21.15.6:0/3717086511 learned_addr learned my addr 172.21.15.6:0/3717086511 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:12.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): get_auth_request con 0x7f4990098670 auth_method 0 2023-09-30T00:45:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): get_auth_request con 0x7f4990098e90 auth_method 0 2023-09-30T00:45:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:12.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4990098a40 0x7f4990141950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:12.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f499662b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995e2a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 10 monclient(hunting): get_auth_request con 0x7f4990098a40 auth_method 0 2023-09-30T00:45:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:12.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:12.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:12.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:12.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.143+0000 7f4995629700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:12.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f4995629700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:12.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 10 monclient(hunting): handle_auth_done global_id 4385 payload 824 2023-09-30T00:45:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 1 -- 172.21.15.6:0/3717086511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4990098a40 msgr2=0x7f4990141950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4990098a40 0x7f4990141950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 1 -- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 msgr2=0x7f499009efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 0x7f499009efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 10 monclient: _finish_hunting 0 2023-09-30T00:45:12.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 1 monclient: found mon.noname-c 2023-09-30T00:45:12.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:12.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:12.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f499662b700 1 -- 172.21.15.6:0/3717086511 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f498c009420 con 0x7f4990098e90 2023-09-30T00:45:12.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.144+0000 7f4995e2a700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 0x7f499009efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:12.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499662b700 1 -- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 msgr2=0x7f4990141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499662b700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49900a0650 tx=0x7f4988005340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f498800a9c0 con 0x7f4990098e90 2023-09-30T00:45:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:12.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: dump: 2023-09-30T00:45:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:12.189 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:12.190 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: _finish_auth 0 2023-09-30T00:45:12.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 5 monclient: authenticate success, global_id 4385 2023-09-30T00:45:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f498800ab40 con 0x7f4990098e90 2023-09-30T00:45:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 10 monclient: _renew_subs 2023-09-30T00:45:12.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49880070f0 con 0x7f4990098e90 2023-09-30T00:45:12.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:12.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f499009d150 con 0x7f4990098e90 2023-09-30T00:45:12.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:12.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:12.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4987fff700 4 set_mon_vals no callback set 2023-09-30T00:45:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f4985ffb700 10 monclient: dump: 2023-09-30T00:45:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.193 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:12.194 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:12.194 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 10 monclient: _renew_subs 2023-09-30T00:45:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.146+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f499009f500 con 0x7f4990098e90 2023-09-30T00:45:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f499662b700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49900a0650 tx=0x7f4988005340 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f4988047b60 osd_map(17..17 src has 1..17) v4 2023-09-30T00:45:12.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f49880074a0 con 0x7f4990098e90 2023-09-30T00:45:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f4985ffb700 1 --2- 172.21.15.6:0/3717086511 >> v2:172.21.15.22:6812/142603 conn(0x7f49740357a0 0x7f4974037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f499808e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f499808e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f499808e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:12.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.147+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4990072b00 con 0x7f4990098e90 2023-09-30T00:45:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.148+0000 7f4995e2a700 1 --2- 172.21.15.6:0/3717086511 >> v2:172.21.15.22:6812/142603 conn(0x7f49740357a0 0x7f4974037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.148+0000 7f4995e2a700 10 monclient: get_auth_request con 0x7f49740357a0 auth_method 0 2023-09-30T00:45:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.150+0000 7f4995e2a700 1 --2- 172.21.15.6:0/3717086511 >> v2:172.21.15.22:6812/142603 conn(0x7f49740357a0 0x7f4974037c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f498c00a010 tx=0x7f498c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:12.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.152+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4988047b60 con 0x7f4990098e90 2023-09-30T00:45:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.976+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4988010d40 con 0x7f4990098e90 2023-09-30T00:45:12.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.976+0000 7f4985ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:12.977 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:12.976+0000 7f4985ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f499808e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f499808e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f499808e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f499009e3e0 con 0x7f4990098e90 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f4985ffb700 1 -- 172.21.15.6:0/3717086511 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4988029080 con 0x7f4990098e90 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f4985ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.226+0000 7f4985ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:13.228 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:13.228 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 >> v2:172.21.15.22:6812/142603 conn(0x7f49740357a0 msgr2=0x7f4974037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 --2- 172.21.15.6:0/3717086511 >> v2:172.21.15.22:6812/142603 conn(0x7f49740357a0 0x7f4974037c50 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f498c00a010 tx=0x7f498c016000 comp rx=0 tx=0).stop 2023-09-30T00:45:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 10 monclient: shutdown 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 msgr2=0x7f4990141e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f49900a0650 tx=0x7f4988005340 comp rx=0 tx=0).stop 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 shutdown_connections 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 --2- 172.21.15.6:0/3717086511 >> v2:172.21.15.22:6812/142603 conn(0x7f49740357a0 0x7f4974037c50 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4990098e90 0x7f4990141e90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4990098670 0x7f499009efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 --2- 172.21.15.6:0/3717086511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4990098a40 0x7f4990141950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 >> 172.21.15.6:0/3717086511 conn(0x7f49900a5480 msgr2=0x7f4990133ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 shutdown_connections 2023-09-30T00:45:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:13.228+0000 7f499808e700 1 -- 172.21.15.6:0/3717086511 wait complete. 2023-09-30T00:45:16.243 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.381+0000 7f61e46e4700 10 monclient: get_monmap_and_config 2023-09-30T00:45:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.381+0000 7f61e46e4700 10 monclient: build_initial_monmap 2023-09-30T00:45:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.381+0000 7f61e46e4700 10 monclient: monmap: 2023-09-30T00:45:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:16.382 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:16.382387+0000 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:16.382387+0000 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.381+0000 7f61e46e4700 1 Processor -- start 2023-09-30T00:45:16.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.381+0000 7f61e46e4700 1 -- start start 2023-09-30T00:45:16.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.381+0000 7f61e46e4700 10 monclient: init 2023-09-30T00:45:16.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:16.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f61dc098670 0x7f61dc132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): picked mon.noname-c con 0x7f61dc098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): picked mon.noname-b con 0x7f61dc098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f61dc098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61dc0a1000 con 0x7f61dc098e90 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f61dc0a0a50 con 0x7f61dc098670 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:16.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc0a0730 con 0x7f61dc098a40 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e46e4700 10 monclient(hunting): authenticate will time out at 1312.680602s 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51592/0 (socket says 172.21.15.6:51592) 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 1 -- 172.21.15.6:0/3211062020 learned_addr learned my addr 172.21.15.6:0/3211062020 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): get_auth_request con 0x7f61dc098a40 auth_method 0 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 1 --2- 172.21.15.6:0/3211062020 >> v2:172.21.15.22:3300/0 conn(0x7f61dc098670 0x7f61dc132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): get_auth_request con 0x7f61dc098670 auth_method 0 2023-09-30T00:45:16.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.382+0000 7f61e2480700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_done global_id 4388 payload 274 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 1 -- 172.21.15.6:0/3211062020 >> v2:172.21.15.22:3300/0 conn(0x7f61dc098670 msgr2=0x7f61dc132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:16.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 1 --2- 172.21.15.6:0/3211062020 >> v2:172.21.15.22:3300/0 conn(0x7f61dc098670 0x7f61dc132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 1 -- 172.21.15.6:0/3211062020 >> v1:172.21.15.22:6789/0 conn(0x7f61dc098e90 legacy=0x7f61dc12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 10 monclient: _finish_hunting 0 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 1 monclient: found mon.noname-b 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.383+0000 7f61e1c7f700 1 -- 172.21.15.6:0/3211062020 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61dc068df0 con 0x7f61dc098a40 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e1c7f700 1 -- 172.21.15.6:0/3211062020 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 msgr2=0x7f61dc12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e1c7f700 1 --2- 172.21.15.6:0/3211062020 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12ede0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0099b0 tx=0x7f61cc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2aa63d236590b57 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 1 -- 172.21.15.6:0/3211062020 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc009ea0 con 0x7f61dc098a40 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:16.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: dump: 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:16.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: _finish_auth 0 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 1 -- 172.21.15.6:0/3211062020 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc00ba30 con 0x7f61dc098a40 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e46e4700 5 monclient: authenticate success, global_id 4388 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e46e4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e46e4700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 1 -- 172.21.15.6:0/3211062020 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc00bdf0 con 0x7f61dc098a40 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e46e4700 4 set_mon_vals no callback set 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:16.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.385+0000 7f61e147e700 10 monclient: dump: 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 10 monclient: shutdown 2023-09-30T00:45:16.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 -- 172.21.15.6:0/3211062020 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 msgr2=0x7f61dc12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 --2- 172.21.15.6:0/3211062020 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12ede0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f61cc0099b0 tx=0x7f61cc009250 comp rx=0 tx=0).stop 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 -- 172.21.15.6:0/3211062020 shutdown_connections 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 --2- 172.21.15.6:0/3211062020 >> v2:172.21.15.22:3300/0 conn(0x7f61dc098670 0x7f61dc132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 --2- 172.21.15.6:0/3211062020 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12ede0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 -- 172.21.15.6:0/3211062020 >> 172.21.15.6:0/3211062020 conn(0x7f61dc0a5480 msgr2=0x7f61dc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 -- 172.21.15.6:0/3211062020 shutdown_connections 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 1 -- 172.21.15.6:0/3211062020 wait complete. 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 10 monclient: build_initial_monmap 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.386+0000 7f61e46e4700 10 monclient: monmap: 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:16.395 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 Processor -- start 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 -- start start 2023-09-30T00:45:16.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient: init 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 0x7f61dc141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): picked mon.noname-a con 0x7f61dc098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): picked mon.noname-c con 0x7f61dc098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61dc098e90 0x7f61dc09c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:16.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): picked mon.noname-b con 0x7f61dc098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc135390 con 0x7f61dc098670 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc12da70 con 0x7f61dc098e90 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f61dc1429e0 con 0x7f61dc098a40 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e46e4700 10 monclient(hunting): authenticate will time out at 1312.685692s 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 0x7f61dc141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 0x7f61dc141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:38792/0 (socket says 172.21.15.6:38792) 2023-09-30T00:45:16.401 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 1 -- 172.21.15.6:0/1501356467 learned_addr learned my addr 172.21.15.6:0/1501356467 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): get_auth_request con 0x7f61dc098670 auth_method 0 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): get_auth_request con 0x7f61dc098a40 auth_method 0 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:16.402 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2480700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.387+0000 7f61e2c81700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61dc098e90 0x7f61dc09c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): get_auth_request con 0x7f61dc098e90 auth_method 0 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2c81700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:16.403 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 10 monclient(hunting): handle_auth_done global_id 4391 payload 824 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 1 -- 172.21.15.6:0/1501356467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61dc098e90 msgr2=0x7f61dc09c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61dc098e90 0x7f61dc09c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 1 -- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 msgr2=0x7f61dc141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 0x7f61dc141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 10 monclient: _finish_hunting 0 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 1 monclient: found mon.noname-c 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e1c7f700 1 -- 172.21.15.6:0/1501356467 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61cc0094f0 con 0x7f61dc098a40 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.388+0000 7f61e2480700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 0x7f61dc141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:16.404 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e1c7f700 1 -- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 msgr2=0x7f61dc12f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e1c7f700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12f1f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61cc002000 tx=0x7f61cc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc0118b0 con 0x7f61dc098a40 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: dump: 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:16.405 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: _finish_auth 0 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 5 monclient: authenticate success, global_id 4391 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61cc020070 con 0x7f61dc098a40 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61cc011ca0 con 0x7f61dc098a40 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 10 monclient: _renew_subs 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:16.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61dc09efb0 con 0x7f61dc098a40 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e0c7d700 4 set_mon_vals no callback set 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61d27fc700 10 monclient: dump: 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:16.407 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 10 monclient: _renew_subs 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.390+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61dc12d4d0 con 0x7f61dc098a40 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f61cc022030 con 0x7f61dc098a40 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61e46e4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61e46e4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61e46e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61dc12de00 con 0x7f61dc098a40 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61d27fc700 1 --2- 172.21.15.6:0/1501356467 >> v2:172.21.15.22:6812/142603 conn(0x7f61c00357f0 0x7f61c0037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.391+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f61cc0495a0 con 0x7f61dc098a40 2023-09-30T00:45:16.408 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.392+0000 7f61e2480700 1 --2- 172.21.15.6:0/1501356467 >> v2:172.21.15.22:6812/142603 conn(0x7f61c00357f0 0x7f61c0037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.392+0000 7f61e2480700 10 monclient: get_auth_request con 0x7f61c00357f0 auth_method 0 2023-09-30T00:45:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.394+0000 7f61e2480700 1 --2- 172.21.15.6:0/1501356467 >> v2:172.21.15.22:6812/142603 conn(0x7f61c00357f0 0x7f61c0037ca0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009fd0 tx=0x7f61d80091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.396+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f61dc12de00 con 0x7f61dc098a40 2023-09-30T00:45:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.396+0000 7f61d27fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:16.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.396+0000 7f61d27fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.646+0000 7f61e46e4700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.646+0000 7f61e46e4700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.646+0000 7f61e46e4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.646+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f61dc12de00 con 0x7f61dc098a40 2023-09-30T00:45:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.647+0000 7f61d27fc700 1 -- 172.21.15.6:0/1501356467 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f61dc12de00 con 0x7f61dc098a40 2023-09-30T00:45:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.647+0000 7f61d27fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:16.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.647+0000 7f61d27fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:16.649 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:16.649 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:16.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 >> v2:172.21.15.22:6812/142603 conn(0x7f61c00357f0 msgr2=0x7f61c0037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 --2- 172.21.15.6:0/1501356467 >> v2:172.21.15.22:6812/142603 conn(0x7f61c00357f0 0x7f61c0037ca0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f61d8009fd0 tx=0x7f61d80091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 10 monclient: shutdown 2023-09-30T00:45:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:16.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 msgr2=0x7f61dc12f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12f1f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f61cc002000 tx=0x7f61cc016000 comp rx=0 tx=0).stop 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 shutdown_connections 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 --2- 172.21.15.6:0/1501356467 >> v2:172.21.15.22:6812/142603 conn(0x7f61c00357f0 0x7f61c0037ca0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f61dc098e90 0x7f61dc09c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f61dc098670 0x7f61dc141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 --2- 172.21.15.6:0/1501356467 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f61dc098a40 0x7f61dc12f1f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 >> 172.21.15.6:0/1501356467 conn(0x7f61dc0a5480 msgr2=0x7f61dc069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 shutdown_connections 2023-09-30T00:45:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:16.649+0000 7f61e46e4700 1 -- 172.21.15.6:0/1501356467 wait complete. 2023-09-30T00:45:19.664 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.802+0000 7f6fdd6dc700 10 monclient: get_monmap_and_config 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.802+0000 7f6fdd6dc700 10 monclient: build_initial_monmap 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.802+0000 7f6fdd6dc700 10 monclient: monmap: 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:19.803580+0000 2023-09-30T00:45:19.803 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:19.803580+0000 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:19.804 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.802+0000 7f6fdd6dc700 1 Processor -- start 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 1 -- start start 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient: init 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient: _add_conns ranks=[0,1,4,3,2] 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6fd8098600 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:19.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6fd8132410 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6fd8138500 0x7f6fd813a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient(hunting): picked mon.noname-c con 0x7f6fd8138500 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd80a0f60 con 0x7f6fd8098600 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6fd80990e0 con 0x7f6fd8132410 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6fd806a280 con 0x7f6fd8138500 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fdd6dc700 10 monclient(hunting): authenticate will time out at 1316.101667s 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd6ffd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6fd8098600 0x7f6fd80989d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49016/0 (socket says 172.21.15.6:49016) 2023-09-30T00:45:19.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd6ffd700 1 -- 172.21.15.6:0/3612169821 learned_addr learned my addr 172.21.15.6:0/3612169821 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd77fe700 1 --2- 172.21.15.6:0/3612169821 >> v2:172.21.15.22:3300/0 conn(0x7f6fd8138500 0x7f6fd813a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd77fe700 10 monclient(hunting): get_auth_request con 0x7f6fd8138500 auth_method 0 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.803+0000 7f6fd77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_done global_id 4408 payload 274 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3612169821 >> v1:172.21.15.22:6789/0 conn(0x7f6fd8132410 legacy=0x7f6fd8132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:19.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3612169821 >> v1:172.21.15.6:6789/0 conn(0x7f6fd8098600 legacy=0x7f6fd80989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 10 monclient: _finish_hunting 0 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 1 monclient: found mon.noname-c 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.804+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3612169821 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd80a1cb0 con 0x7f6fd8138500 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.806+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3612169821 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8138500 msgr2=0x7f6fd813a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:19.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.806+0000 7f6fd77fe700 1 --2- 172.21.15.6:0/3612169821 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8138500 0x7f6fd813a8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009e60 tx=0x7f6fd0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b14e6b346194a4a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:19.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 1 -- 172.21.15.6:0/3612169821 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000e070 con 0x7f6fd8138500 2023-09-30T00:45:19.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:19.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: dump: 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:19.817 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: _finish_auth 0 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 1 -- 172.21.15.6:0/3612169821 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd0012430 con 0x7f6fd8138500 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 1 -- 172.21.15.6:0/3612169821 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd000bea0 con 0x7f6fd8138500 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fd5ffb700 10 monclient: dump: 2023-09-30T00:45:19.818 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 5 monclient: authenticate success, global_id 4408 2023-09-30T00:45:19.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 4 set_mon_vals no callback set 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 10 monclient: shutdown 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3612169821 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8138500 msgr2=0x7f6fd813a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.807+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3612169821 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8138500 0x7f6fd813a8e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0009e60 tx=0x7f6fd0009780 comp rx=0 tx=0).stop 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3612169821 shutdown_connections 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3612169821 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8138500 0x7f6fd813a8e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3612169821 >> 172.21.15.6:0/3612169821 conn(0x7f6fd80a54c0 msgr2=0x7f6fd80a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3612169821 shutdown_connections 2023-09-30T00:45:19.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3612169821 wait complete. 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 10 monclient: build_initial_monmap 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.808+0000 7f6fdd6dc700 10 monclient: monmap: 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:19.821 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 Processor -- start 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 -- start start 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient: init 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8098600 0x7f6fd809cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): picked mon.noname-b con 0x7f6fd8098600 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 0x7f6fd809d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:19.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): picked mon.noname-a con 0x7f6fd8132410 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 0x7f6fd809d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): picked mon.noname-c con 0x7f6fd8138500 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80a3cd0 con 0x7f6fd8132410 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd81335d0 con 0x7f6fd8098600 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6fd80a1460 con 0x7f6fd8138500 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:19.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fdd6dc700 10 monclient(hunting): authenticate will time out at 1316.107673s 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 0x7f6fd809d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 0x7f6fd809d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 0x7f6fd809d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41570/0 (socket says 172.21.15.6:41570) 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 1 -- 172.21.15.6:0/3001020809 learned_addr learned my addr 172.21.15.6:0/3001020809 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): get_auth_request con 0x7f6fd8132410 auth_method 0 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): get_auth_request con 0x7f6fd8138500 auth_method 0 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:19.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd6ffd700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8098600 0x7f6fd809cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd6ffd700 10 monclient(hunting): get_auth_request con 0x7f6fd8098600 auth_method 0 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:19.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.809+0000 7f6fd67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 10 monclient(hunting): handle_auth_done global_id 4394 payload 824 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3001020809 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8098600 msgr2=0x7f6fd809cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8098600 0x7f6fd809cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 msgr2=0x7f6fd809d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:19.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 0x7f6fd809d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 10 monclient: _finish_hunting 0 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 1 monclient: found mon.noname-c 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3001020809 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fd0009a20 con 0x7f6fd8138500 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.810+0000 7f6fd67fc700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 0x7f6fd809d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.812+0000 7f6fd77fe700 1 -- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 msgr2=0x7f6fd809d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.812+0000 7f6fd77fe700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 0x7f6fd809d710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0003000 tx=0x7f6fd0018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.812+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0012e70 con 0x7f6fd8138500 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.812+0000 7f6fbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:19.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: dump: 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:19.828 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: _finish_auth 0 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6fd0021070 con 0x7f6fd8138500 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 5 monclient: authenticate success, global_id 4394 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 10 monclient: _renew_subs 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fd81332f0 con 0x7f6fd8138500 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6fd0026680 con 0x7f6fd8138500 2023-09-30T00:45:19.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fbeffd700 10 monclient: dump: 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:19.830 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 10 monclient: _renew_subs 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fd57fa700 4 set_mon_vals no callback set 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.813+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fd809e870 con 0x7f6fd8138500 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6fd0023030 con 0x7f6fd8138500 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fdd6dc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fdd6dc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fdd6dc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:19.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fd8133db0 con 0x7f6fd8138500 2023-09-30T00:45:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fbeffd700 1 --2- 172.21.15.6:0/3001020809 >> v2:172.21.15.22:6812/142603 conn(0x7f6fc40357a0 0x7f6fc4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6fd0049b10 con 0x7f6fd8138500 2023-09-30T00:45:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fd6ffd700 1 --2- 172.21.15.6:0/3001020809 >> v2:172.21.15.22:6812/142603 conn(0x7f6fc40357a0 0x7f6fc4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.814+0000 7f6fd6ffd700 10 monclient: get_auth_request con 0x7f6fc40357a0 auth_method 0 2023-09-30T00:45:19.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.817+0000 7f6fd6ffd700 1 --2- 172.21.15.6:0/3001020809 >> v2:172.21.15.22:6812/142603 conn(0x7f6fc40357a0 0x7f6fc4037c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8007880 tx=0x7f6fc8007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:19.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.819+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6fd001c420 con 0x7f6fd8138500 2023-09-30T00:45:19.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.819+0000 7f6fbeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:19.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:19.819+0000 7f6fbeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.068+0000 7f6fdd6dc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.068+0000 7f6fdd6dc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.068+0000 7f6fdd6dc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.068+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6fd8133db0 con 0x7f6fd8138500 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.069+0000 7f6fbeffd700 1 -- 172.21.15.6:0/3001020809 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6fd001b3a0 con 0x7f6fd8138500 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.069+0000 7f6fbeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.069+0000 7f6fbeffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:20.070 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:20.071 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 >> v2:172.21.15.22:6812/142603 conn(0x7f6fc40357a0 msgr2=0x7f6fc4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3001020809 >> v2:172.21.15.22:6812/142603 conn(0x7f6fc40357a0 0x7f6fc4037c50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6fc8007880 tx=0x7f6fc8007f90 comp rx=0 tx=0).stop 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 10 monclient: shutdown 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 msgr2=0x7f6fd809d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 0x7f6fd809d710 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f6fd0003000 tx=0x7f6fd0018000 comp rx=0 tx=0).stop 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 shutdown_connections 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3001020809 >> v2:172.21.15.22:6812/142603 conn(0x7f6fc40357a0 0x7f6fc4037c50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6fd8138500 0x7f6fd809d710 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6fd8098600 0x7f6fd809cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 --2- 172.21.15.6:0/3001020809 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6fd8132410 0x7f6fd809d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 >> 172.21.15.6:0/3001020809 conn(0x7f6fd80a54c0 msgr2=0x7f6fd8137b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 shutdown_connections 2023-09-30T00:45:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:20.071+0000 7f6fdd6dc700 1 -- 172.21.15.6:0/3001020809 wait complete. 2023-09-30T00:45:23.086 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:23.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.223+0000 7f879f75d700 10 monclient: get_monmap_and_config 2023-09-30T00:45:23.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.223+0000 7f879f75d700 10 monclient: build_initial_monmap 2023-09-30T00:45:23.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.223+0000 7f879f75d700 10 monclient: monmap: 2023-09-30T00:45:23.224 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:23.224 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:23.224 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:23.224481+0000 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:23.224481+0000 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:23.225 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:23.226 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:23.226 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.223+0000 7f879f75d700 1 Processor -- start 2023-09-30T00:45:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 1 -- start start 2023-09-30T00:45:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient: init 2023-09-30T00:45:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:23.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient: _add_conns ranks=[4,3,0,2,1] 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8798098670 0x7f8798132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): picked mon.noname-c con 0x7f8798098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8798098a40 0x7f879812ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): picked mon.noname-a con 0x7f8798098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8798098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87980a1000 con 0x7f8798098e90 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980a0a50 con 0x7f8798098a40 2023-09-30T00:45:23.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f87980a0730 con 0x7f8798098670 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879f75d700 10 monclient(hunting): authenticate will time out at 1319.522779s 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8798098a40 0x7f879812ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8798098a40 0x7f879812ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41584/0 (socket says 172.21.15.6:41584) 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879dcfa700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8798098e90 0x7f879812f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:49018/0 (socket says 172.21.15.6:49018) 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 1 -- 172.21.15.6:0/1457748928 learned_addr learned my addr 172.21.15.6:0/1457748928 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 10 monclient(hunting): get_auth_request con 0x7f8798098a40 auth_method 0 2023-09-30T00:45:23.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879ccf8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.224+0000 7f879d4f9700 1 --2- 172.21.15.6:0/1457748928 >> v2:172.21.15.22:3300/0 conn(0x7f8798098670 0x7f8798132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): get_auth_request con 0x7f8798098670 auth_method 0 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:23.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879d4f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 10 monclient(hunting): handle_auth_done global_id 4395 payload 274 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 1 -- 172.21.15.6:0/1457748928 >> v2:172.21.15.22:3300/0 conn(0x7f8798098670 msgr2=0x7f8798132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 1 --2- 172.21.15.6:0/1457748928 >> v2:172.21.15.22:3300/0 conn(0x7f8798098670 0x7f8798132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 1 -- 172.21.15.6:0/1457748928 >> v1:172.21.15.6:6789/0 conn(0x7f8798098e90 legacy=0x7f879812f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 10 monclient: _finish_hunting 0 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 1 monclient: found mon.noname-a 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.225+0000 7f879ccf8700 1 -- 172.21.15.6:0/1457748928 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8798068df0 con 0x7f8798098a40 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.227+0000 7f879ccf8700 1 -- 172.21.15.6:0/1457748928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098a40 msgr2=0x7f879812ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:23.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.227+0000 7f879ccf8700 1 --2- 172.21.15.6:0/1457748928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098a40 0x7f879812ede0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f879809fe60 tx=0x7f8788009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=524800dca142703c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 1 -- 172.21.15.6:0/1457748928 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8788009e70 con 0x7f8798098a40 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: dump: 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:23.241 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: _finish_auth 0 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:23.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 5 monclient: authenticate success, global_id 4395 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 1 -- 172.21.15.6:0/1457748928 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f878800ba30 con 0x7f8798098a40 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 1 -- 172.21.15.6:0/1457748928 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f878800bdf0 con 0x7f8798098a40 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f878ffff700 10 monclient: dump: 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:23.243 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 4 set_mon_vals no callback set 2023-09-30T00:45:23.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 10 monclient: shutdown 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 1 -- 172.21.15.6:0/1457748928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098a40 msgr2=0x7f879812ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.232+0000 7f879f75d700 1 --2- 172.21.15.6:0/1457748928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098a40 0x7f879812ede0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f879809fe60 tx=0x7f8788009250 comp rx=0 tx=0).stop 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 1 -- 172.21.15.6:0/1457748928 shutdown_connections 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 1 --2- 172.21.15.6:0/1457748928 >> v2:172.21.15.22:3300/0 conn(0x7f8798098670 0x7f8798132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 1 --2- 172.21.15.6:0/1457748928 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098a40 0x7f879812ede0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 1 -- 172.21.15.6:0/1457748928 >> 172.21.15.6:0/1457748928 conn(0x7f87980a5480 msgr2=0x7f87980a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 1 -- 172.21.15.6:0/1457748928 shutdown_connections 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 1 -- 172.21.15.6:0/1457748928 wait complete. 2023-09-30T00:45:23.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 10 monclient: build_initial_monmap 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.233+0000 7f879f75d700 10 monclient: monmap: 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:23.246 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 Processor -- start 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 -- start start 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient: init 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 0x7f8798141930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): picked mon.noname-a con 0x7f8798098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8798098a40 0x7f879812c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): picked mon.noname-b con 0x7f8798098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:23.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8798098e90 0x7f879809cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): picked mon.noname-c con 0x7f8798098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879811ccd0 con 0x7f8798098670 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f879811c430 con 0x7f8798098a40 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f879811c710 con 0x7f8798098e90 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879f75d700 10 monclient(hunting): authenticate will time out at 1319.532854s 2023-09-30T00:45:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.234+0000 7f879d4f9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 0x7f8798141930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8798098e90 0x7f879809cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 0x7f8798141930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41592/0 (socket says 172.21.15.6:41592) 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 -- 172.21.15.6:0/1905402896 learned_addr learned my addr 172.21.15.6:0/1905402896 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): get_auth_request con 0x7f8798098670 auth_method 0 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): get_auth_request con 0x7f8798098e90 auth_method 0 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8798098a40 0x7f879812c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879dcfa700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): get_auth_request con 0x7f8798098a40 auth_method 0 2023-09-30T00:45:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879ccf8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient(hunting): handle_auth_done global_id 4398 payload 824 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 -- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8798098e90 msgr2=0x7f879809cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8798098e90 0x7f879809cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 -- 172.21.15.6:0/1905402896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8798098a40 msgr2=0x7f879812c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:23.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8798098a40 0x7f879812c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient: _finish_hunting 0 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 monclient: found mon.noname-a 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.235+0000 7f879d4f9700 1 -- 172.21.15.6:0/1905402896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87880094f0 con 0x7f8798098670 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879d4f9700 1 -- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 msgr2=0x7f8798141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879d4f9700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 0x7f8798141930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f879400ec80 tx=0x7f87940091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8794009690 con 0x7f8798098670 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:23.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: dump: 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:23.253 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: _finish_auth 0 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f879400ca60 con 0x7f8798098670 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 5 monclient: authenticate success, global_id 4398 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 10 monclient: _renew_subs 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f879811c9f0 con 0x7f8798098670 2023-09-30T00:45:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f879400ce20 con 0x7f8798098670 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878d7fa700 10 monclient: dump: 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f878f7fe700 4 set_mon_vals no callback set 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 10 monclient: _renew_subs 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.237+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f879809f370 con 0x7f8798098670 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.238+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8794013680 con 0x7f8798098670 2023-09-30T00:45:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.238+0000 7f879f75d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.238+0000 7f879f75d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.238+0000 7f879f75d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.238+0000 7f878d7fa700 1 --2- 172.21.15.6:0/1905402896 >> v2:172.21.15.22:6812/142603 conn(0x7f877c0357a0 0x7f877c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.238+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f879809f5c0 con 0x7f8798098670 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.239+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8794047c00 con 0x7f8798098670 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.239+0000 7f879ccf8700 1 --2- 172.21.15.6:0/1905402896 >> v2:172.21.15.22:6812/142603 conn(0x7f877c0357a0 0x7f877c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.239+0000 7f879ccf8700 10 monclient: get_auth_request con 0x7f877c0357a0 auth_method 0 2023-09-30T00:45:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.241+0000 7f879ccf8700 1 --2- 172.21.15.6:0/1905402896 >> v2:172.21.15.22:6812/142603 conn(0x7f877c0357a0 0x7f877c037c50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f879809fe60 tx=0x7f8788016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.244+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f879809f5c0 con 0x7f8798098670 2023-09-30T00:45:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.244+0000 7f878d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.244+0000 7f878d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.488+0000 7f879f75d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.488+0000 7f879f75d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.488+0000 7f879f75d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.488+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f879809f5c0 con 0x7f8798098670 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.489+0000 7f878d7fa700 1 -- 172.21.15.6:0/1905402896 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f879809f5c0 con 0x7f8798098670 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.489+0000 7f878d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.489+0000 7f878d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:23.490 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:23.491 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 >> v2:172.21.15.22:6812/142603 conn(0x7f877c0357a0 msgr2=0x7f877c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 --2- 172.21.15.6:0/1905402896 >> v2:172.21.15.22:6812/142603 conn(0x7f877c0357a0 0x7f877c037c50 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f879809fe60 tx=0x7f8788016000 comp rx=0 tx=0).stop 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 10 monclient: shutdown 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 msgr2=0x7f8798141930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 0x7f8798141930 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f879400ec80 tx=0x7f87940091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 shutdown_connections 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 --2- 172.21.15.6:0/1905402896 >> v2:172.21.15.22:6812/142603 conn(0x7f877c0357a0 0x7f877c037c50 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8798098e90 0x7f879809cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8798098670 0x7f8798141930 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 --2- 172.21.15.6:0/1905402896 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8798098a40 0x7f879812c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 >> 172.21.15.6:0/1905402896 conn(0x7f87980a5480 msgr2=0x7f87981308f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 shutdown_connections 2023-09-30T00:45:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:23.491+0000 7f879f75d700 1 -- 172.21.15.6:0/1905402896 wait complete. 2023-09-30T00:45:26.506 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:26.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.644+0000 7f6451b4b700 10 monclient: get_monmap_and_config 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.644+0000 7f6451b4b700 10 monclient: build_initial_monmap 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.644+0000 7f6451b4b700 10 monclient: monmap: 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:26.645375+0000 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:26.645375+0000 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:26.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:26.646 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:26.646 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.644+0000 7f6451b4b700 1 Processor -- start 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 1 -- start start 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient: init 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient: _add_conns ranks=[1,3,2,4,0] 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f644c098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:26.649 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f644c132d00 0x7f644c137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): picked mon.noname-a con 0x7f644c132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c137b60 0x7f644c139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): picked mon.noname-b con 0x7f644c137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f644c0a1000 con 0x7f644c098670 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f644c099150 con 0x7f644c132d00 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c0a0a50 con 0x7f644c137b60 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:26.650 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f6451b4b700 10 monclient(hunting): authenticate will time out at 1322.943653s 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f644c132d00 0x7f644c137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c137b60 0x7f644c139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f644c132d00 0x7f644c137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42082/0 (socket says 172.21.15.6:42082) 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 1 -- 172.21.15.6:0/1001156719 learned_addr learned my addr 172.21.15.6:0/1001156719 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): get_auth_request con 0x7f644c132d00 auth_method 0 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): get_auth_request con 0x7f644c137b60 auth_method 0 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:26.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.645+0000 7f644bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 10 monclient(hunting): handle_auth_done global_id 4401 payload 274 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 1 -- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c137b60 msgr2=0x7f644c139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 1 --2- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c137b60 0x7f644c139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 1 -- 172.21.15.6:0/1001156719 >> v1:172.21.15.22:6789/0 conn(0x7f644c098670 legacy=0x7f644c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:26.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 10 monclient: _finish_hunting 0 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 1 monclient: found mon.noname-a 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644affd700 1 -- 172.21.15.6:0/1001156719 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f644c0a0730 con 0x7f644c132d00 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.646+0000 7f644bfff700 1 --2- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c137b60 0x7f644c139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644affd700 1 -- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c132d00 msgr2=0x7f644c137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644affd700 1 --2- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c132d00 0x7f644c137620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64400099e0 tx=0x7f6440009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3777e467e289fa42 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 1 -- 172.21.15.6:0/1001156719 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f644000b9f0 con 0x7f644c132d00 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: dump: 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:26.653 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:26.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f6451b4b700 5 monclient: authenticate success, global_id 4401 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 1 -- 172.21.15.6:0/1001156719 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f644000bb50 con 0x7f644c132d00 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f6451b4b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f6451b4b700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 1 -- 172.21.15.6:0/1001156719 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6440011420 con 0x7f644c132d00 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f6451b4b700 4 set_mon_vals no callback set 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.648+0000 7f644a7fc700 10 monclient: dump: 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:26.655 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:26.656 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 10 monclient: shutdown 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 -- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c132d00 msgr2=0x7f644c137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 --2- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c132d00 0x7f644c137620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f64400099e0 tx=0x7f6440009250 comp rx=0 tx=0).stop 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 -- 172.21.15.6:0/1001156719 shutdown_connections 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 --2- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c137b60 0x7f644c139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:26.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 --2- 172.21.15.6:0/1001156719 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c132d00 0x7f644c137620 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 -- 172.21.15.6:0/1001156719 >> 172.21.15.6:0/1001156719 conn(0x7f644c0a5480 msgr2=0x7f644c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 -- 172.21.15.6:0/1001156719 shutdown_connections 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.649+0000 7f6451b4b700 1 -- 172.21.15.6:0/1001156719 wait complete. 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.650+0000 7f6451b4b700 10 monclient: build_initial_monmap 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.650+0000 7f6451b4b700 10 monclient: monmap: 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:26.662 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.650+0000 7f6451b4b700 1 Processor -- start 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.650+0000 7f6451b4b700 1 -- start start 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.650+0000 7f6451b4b700 10 monclient: init 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): picked mon.noname-c con 0x7f644c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f644c132d00 0x7f644c134a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:26.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): picked mon.noname-b con 0x7f644c132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c137b60 0x7f644c134f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): picked mon.noname-a con 0x7f644c137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c131460 con 0x7f644c137b60 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f644c137730 con 0x7f644c132d00 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f644c12d7a0 con 0x7f644c098670 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f6451b4b700 10 monclient(hunting): authenticate will time out at 1322.949555s 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:26.664 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51638/0 (socket says 172.21.15.6:51638) 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c137b60 0x7f644c134f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 1 -- 172.21.15.6:0/869404798 learned_addr learned my addr 172.21.15.6:0/869404798 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): get_auth_request con 0x7f644c098670 auth_method 0 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): get_auth_request con 0x7f644c137b60 auth_method 0 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f644c132d00 0x7f644c134a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:26.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): get_auth_request con 0x7f644c132d00 auth_method 0 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.651+0000 7f644b7fe700 10 monclient(hunting): handle_auth_done global_id 4403 payload 824 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 1 -- 172.21.15.6:0/869404798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f644c132d00 msgr2=0x7f644c134a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f644c132d00 0x7f644c134a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 1 -- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c137b60 msgr2=0x7f644c134f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c137b60 0x7f644c134f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 1 monclient: found mon.noname-c 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644b7fe700 1 -- 172.21.15.6:0/869404798 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6444009a20 con 0x7f644c098670 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.652+0000 7f644bfff700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c137b60 0x7f644c134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f644b7fe700 1 -- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 msgr2=0x7f644c1344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f644b7fe700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f643c00c5d0 tx=0x7f643c007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c0117a0 con 0x7f644c098670 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: dump: 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: _finish_auth 0 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f643c011920 con 0x7f644c098670 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 5 monclient: authenticate success, global_id 4403 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 10 monclient: _renew_subs 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64400094f0 con 0x7f644c098670 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f643c011ce0 con 0x7f644c098670 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:26.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f64337fe700 10 monclient: dump: 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:26.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6449ffb700 4 set_mon_vals no callback set 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 10 monclient: _renew_subs 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.654+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f644c12d540 con 0x7f644c098670 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f644b7fe700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f643c00c5d0 tx=0x7f643c007f60 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f643c047740 osd_map(17..17 src has 1..17) v4 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f643c012630 con 0x7f644c098670 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f64337fe700 1 --2- 172.21.15.6:0/869404798 >> v2:172.21.15.22:6812/142603 conn(0x7f64380357d0 0x7f6438037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f6451b4b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f6451b4b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f6451b4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f644c072b00 con 0x7f644c098670 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f644affd700 1 --2- 172.21.15.6:0/869404798 >> v2:172.21.15.22:6812/142603 conn(0x7f64380357d0 0x7f6438037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.655+0000 7f644affd700 10 monclient: get_auth_request con 0x7f64380357d0 auth_method 0 2023-09-30T00:45:26.671 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.658+0000 7f644affd700 1 --2- 172.21.15.6:0/869404798 >> v2:172.21.15.22:6812/142603 conn(0x7f64380357d0 0x7f6438037c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f64400099e0 tx=0x7f6440016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:26.672 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:26.660+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f643c047740 con 0x7f644c098670 2023-09-30T00:45:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.484+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f643c011e40 con 0x7f644c098670 2023-09-30T00:45:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.484+0000 7f64337fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:27.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.484+0000 7f64337fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:27.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f6451b4b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f6451b4b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f6451b4b700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f644c0a3ca0 con 0x7f644c098670 2023-09-30T00:45:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f64337fe700 1 -- 172.21.15.6:0/869404798 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f643c02a080 con 0x7f644c098670 2023-09-30T00:45:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f64337fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:27.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.729+0000 7f64337fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:27.731 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:27.731 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 >> v2:172.21.15.22:6812/142603 conn(0x7f64380357d0 msgr2=0x7f6438037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 --2- 172.21.15.6:0/869404798 >> v2:172.21.15.22:6812/142603 conn(0x7f64380357d0 0x7f6438037c80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f64400099e0 tx=0x7f6440016000 comp rx=0 tx=0).stop 2023-09-30T00:45:27.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 10 monclient: shutdown 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 msgr2=0x7f644c1344c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f643c00c5d0 tx=0x7f643c007f60 comp rx=0 tx=0).stop 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 shutdown_connections 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 --2- 172.21.15.6:0/869404798 >> v2:172.21.15.22:6812/142603 conn(0x7f64380357d0 0x7f6438037c80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f644c137b60 0x7f644c134f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f644c098670 0x7f644c1344c0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 --2- 172.21.15.6:0/869404798 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f644c132d00 0x7f644c134a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 >> 172.21.15.6:0/869404798 conn(0x7f644c0a5480 msgr2=0x7f644c0a3590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 shutdown_connections 2023-09-30T00:45:27.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:27.731+0000 7f6451b4b700 1 -- 172.21.15.6:0/869404798 wait complete. 2023-09-30T00:45:30.746 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:30.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.882+0000 7f47cb248700 10 monclient: get_monmap_and_config 2023-09-30T00:45:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.882+0000 7f47cb248700 10 monclient: build_initial_monmap 2023-09-30T00:45:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.883+0000 7f47cb248700 10 monclient: monmap: 2023-09-30T00:45:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:30.884192+0000 2023-09-30T00:45:30.884 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:30.884192+0000 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:30.885 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.883+0000 7f47cb248700 1 Processor -- start 2023-09-30T00:45:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.883+0000 7f47cb248700 1 -- start start 2023-09-30T00:45:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.883+0000 7f47cb248700 10 monclient: init 2023-09-30T00:45:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T00:45:30.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c4132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): picked mon.noname-b con 0x7f47c4098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f47c4098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 0x7f47c412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): picked mon.noname-c con 0x7f47c4098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40a1000 con 0x7f47c4098a40 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c40a0a50 con 0x7f47c4098e90 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c40a0730 con 0x7f47c4098670 2023-09-30T00:45:30.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47cb248700 10 monclient(hunting): authenticate will time out at 1327.182401s 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c3fff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f47c4098a40 0x7f47c412ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:38194/0 (socket says 172.21.15.6:38194) 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c3fff700 1 -- 172.21.15.6:0/1620262181 learned_addr learned my addr 172.21.15.6:0/1620262181 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c4132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): get_auth_request con 0x7f47c4098670 auth_method 0 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:30.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c97e5700 1 --2- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 0x7f47c412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c97e5700 1 -- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 msgr2=0x7f47c412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c97e5700 1 -- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 msgr2=0x7f47c412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c97e5700 1 --2- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 0x7f47c412f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c97e5700 1 --2- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 0x7f47c412f4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:45:30.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_done global_id 4406 payload 274 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 1 -- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 msgr2=0x7f47c412f4f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 1 --2- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 0x7f47c412f4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 1 -- 172.21.15.6:0/1620262181 >> v1:172.21.15.6:6789/0 conn(0x7f47c4098a40 legacy=0x7f47c412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient: _finish_hunting 0 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 1 monclient: found mon.noname-b 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.884+0000 7f47c8fe4700 1 -- 172.21.15.6:0/1620262181 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c4068df0 con 0x7f47c4098670 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c8fe4700 1 -- 172.21.15.6:0/1620262181 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 msgr2=0x7f47c4132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:30.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c8fe4700 1 --2- 172.21.15.6:0/1620262181 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c4132d00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47b40098e0 tx=0x7f47b40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=258db300dd8fa976 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 1 -- 172.21.15.6:0/1620262181 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b400ba30 con 0x7f47c4098670 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: dump: 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:30.897 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: _finish_auth 0 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 5 monclient: authenticate success, global_id 4406 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 1 -- 172.21.15.6:0/1620262181 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b400bb90 con 0x7f47c4098670 2023-09-30T00:45:30.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 4 set_mon_vals no callback set 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 1 -- 172.21.15.6:0/1620262181 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4011420 con 0x7f47c4098670 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47c37fe700 10 monclient: dump: 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:30.899 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 10 monclient: shutdown 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:30.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 1 -- 172.21.15.6:0/1620262181 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 msgr2=0x7f47c4132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 1 --2- 172.21.15.6:0/1620262181 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c4132d00 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f47b40098e0 tx=0x7f47b40091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 1 -- 172.21.15.6:0/1620262181 shutdown_connections 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 1 --2- 172.21.15.6:0/1620262181 >> v2:172.21.15.22:3300/0 conn(0x7f47c4098e90 0x7f47c412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 1 --2- 172.21.15.6:0/1620262181 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c4132d00 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.887+0000 7f47cb248700 1 -- 172.21.15.6:0/1620262181 >> 172.21.15.6:0/1620262181 conn(0x7f47c40a5480 msgr2=0x7f47c40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.888+0000 7f47cb248700 1 -- 172.21.15.6:0/1620262181 shutdown_connections 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.888+0000 7f47cb248700 1 -- 172.21.15.6:0/1620262181 wait complete. 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.888+0000 7f47cb248700 10 monclient: build_initial_monmap 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.888+0000 7f47cb248700 10 monclient: monmap: 2023-09-30T00:45:30.901 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.888+0000 7f47cb248700 1 Processor -- start 2023-09-30T00:45:30.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 -- start start 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient: init 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c412c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): picked mon.noname-c con 0x7f47c4098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47c4098a40 0x7f47c409cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): picked mon.noname-b con 0x7f47c4098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 0x7f47c409d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:30.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): picked mon.noname-a con 0x7f47c4098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c411d240 con 0x7f47c4098e90 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c409eed0 con 0x7f47c4098a40 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c4131300 con 0x7f47c4098670 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47cb248700 10 monclient(hunting): authenticate will time out at 1327.187634s 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c412c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:30.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 0x7f47c409d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c412c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51656/0 (socket says 172.21.15.6:51656) 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 0x7f47c409d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42090/0 (socket says 172.21.15.6:42090) 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 1 -- 172.21.15.6:0/2106364519 learned_addr learned my addr 172.21.15.6:0/2106364519 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): get_auth_request con 0x7f47c4098670 auth_method 0 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47c4098a40 0x7f47c409cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): get_auth_request con 0x7f47c4098e90 auth_method 0 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:30.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): get_auth_request con 0x7f47c4098a40 auth_method 0 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:30.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c97e5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.889+0000 7f47c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 10 monclient(hunting): handle_auth_done global_id 4409 payload 824 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 1 -- 172.21.15.6:0/2106364519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47c4098a40 msgr2=0x7f47c409cd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47c4098a40 0x7f47c409cd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 1 -- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 msgr2=0x7f47c409d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 0x7f47c409d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:30.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 10 monclient: _finish_hunting 0 2023-09-30T00:45:30.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 1 monclient: found mon.noname-c 2023-09-30T00:45:30.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:30.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:30.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c8fe4700 1 -- 172.21.15.6:0/2106364519 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b4009420 con 0x7f47c4098670 2023-09-30T00:45:30.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.890+0000 7f47c97e5700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 0x7f47c409d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c8fe4700 1 -- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 msgr2=0x7f47c412c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c8fe4700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c412c610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f47b400b3c0 tx=0x7f47b4016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b4006af0 con 0x7f47c4098670 2023-09-30T00:45:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:30.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: dump: 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:30.911 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: _finish_auth 0 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4020070 con 0x7f47c4098670 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47cb248700 5 monclient: authenticate success, global_id 4409 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47cb248700 10 monclient: _renew_subs 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47cb248700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c40a1500 con 0x7f47c4098670 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b401b6a0 con 0x7f47c4098670 2023-09-30T00:45:30.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c0ff9700 10 monclient: dump: 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:30.913 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.893+0000 7f47cb248700 10 monclient: _renew_subs 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.892+0000 7f47c2ffd700 4 set_mon_vals no callback set 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.893+0000 7f47cb248700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.893+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c409e8e0 con 0x7f47c4098670 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.893+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f47b4022030 con 0x7f47c4098670 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47c0ff9700 1 --2- 172.21.15.6:0/2106364519 >> v2:172.21.15.22:6812/142603 conn(0x7f47b00357f0 0x7f47b0037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f47b4049760 con 0x7f47c4098670 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47cb248700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:30.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47cb248700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47cb248700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c40983c0 con 0x7f47c4098670 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47c3fff700 1 --2- 172.21.15.6:0/2106364519 >> v2:172.21.15.22:6812/142603 conn(0x7f47b00357f0 0x7f47b0037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.894+0000 7f47c3fff700 10 monclient: get_auth_request con 0x7f47b00357f0 auth_method 0 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.897+0000 7f47c3fff700 1 --2- 172.21.15.6:0/2106364519 >> v2:172.21.15.22:6812/142603 conn(0x7f47b00357f0 0x7f47b0037ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47c409e340 tx=0x7f47b8007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.899+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f47b401c7d0 con 0x7f47c4098670 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.899+0000 7f47c0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:30.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:30.899+0000 7f47c0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:31.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.147+0000 7f47cb248700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:31.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.148+0000 7f47cb248700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:31.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.148+0000 7f47cb248700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:31.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.148+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47c40983c0 con 0x7f47c4098670 2023-09-30T00:45:31.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.148+0000 7f47c0ff9700 1 -- 172.21.15.6:0/2106364519 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f47b401abc0 con 0x7f47c4098670 2023-09-30T00:45:31.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.148+0000 7f47c0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:31.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.148+0000 7f47c0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:31.151 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:31.151 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 >> v2:172.21.15.22:6812/142603 conn(0x7f47b00357f0 msgr2=0x7f47b0037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 --2- 172.21.15.6:0/2106364519 >> v2:172.21.15.22:6812/142603 conn(0x7f47b00357f0 0x7f47b0037ca0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f47c409e340 tx=0x7f47b8007f90 comp rx=0 tx=0).stop 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 10 monclient: shutdown 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 msgr2=0x7f47c412c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c412c610 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f47b400b3c0 tx=0x7f47b4016000 comp rx=0 tx=0).stop 2023-09-30T00:45:31.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 shutdown_connections 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 --2- 172.21.15.6:0/2106364519 >> v2:172.21.15.22:6812/142603 conn(0x7f47b00357f0 0x7f47b0037ca0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f47c4098e90 0x7f47c409d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f47c4098670 0x7f47c412c610 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 --2- 172.21.15.6:0/2106364519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f47c4098a40 0x7f47c409cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 >> 172.21.15.6:0/2106364519 conn(0x7f47c40a5480 msgr2=0x7f47c4130b40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 shutdown_connections 2023-09-30T00:45:31.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:31.150+0000 7f47cb248700 1 -- 172.21.15.6:0/2106364519 wait complete. 2023-09-30T00:45:34.165 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:34.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.303+0000 7fd8201c6700 10 monclient: get_monmap_and_config 2023-09-30T00:45:34.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.303+0000 7fd8201c6700 10 monclient: build_initial_monmap 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.303+0000 7fd8201c6700 10 monclient: monmap: 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:34.304907+0000 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:34.304907+0000 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:34.305 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:34.306 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:34.306 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.304+0000 7fd8201c6700 1 Processor -- start 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.304+0000 7fd8201c6700 1 -- start start 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.304+0000 7fd8201c6700 10 monclient: init 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient: _add_conns ranks=[1,4,2,0,3] 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd818098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fd818132d00 0x7fd818137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): picked mon.noname-c con 0x7fd818132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:34.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 0x7fd818139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): picked mon.noname-b con 0x7fd818137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8180a1000 con 0x7fd818098670 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd818099150 con 0x7fd818132d00 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8180a0a50 con 0x7fd818137b60 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd8201c6700 10 monclient(hunting): authenticate will time out at 1330.603420s 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 0x7fd818139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:34.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 0x7fd818139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51660/0 (socket says 172.21.15.6:51660) 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 -- 172.21.15.6:0/4010374623 learned_addr learned my addr 172.21.15.6:0/4010374623 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): get_auth_request con 0x7fd818137b60 auth_method 0 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 1 --2- 172.21.15.6:0/4010374623 >> v2:172.21.15.22:3300/0 conn(0x7fd818132d00 0x7fd818137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): get_auth_request con 0x7fd818132d00 auth_method 0 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:34.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81d761700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient(hunting): handle_auth_done global_id 4412 payload 274 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 -- 172.21.15.6:0/4010374623 >> v2:172.21.15.22:3300/0 conn(0x7fd818132d00 msgr2=0x7fd818137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 --2- 172.21.15.6:0/4010374623 >> v2:172.21.15.22:3300/0 conn(0x7fd818132d00 0x7fd818137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 -- 172.21.15.6:0/4010374623 >> v1:172.21.15.22:6789/0 conn(0x7fd818098670 legacy=0x7fd818098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient: _finish_hunting 0 2023-09-30T00:45:34.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 monclient: found mon.noname-b 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.305+0000 7fd81e763700 1 -- 172.21.15.6:0/4010374623 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8180a0730 con 0x7fd818137b60 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81e763700 1 -- 172.21.15.6:0/4010374623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 msgr2=0x7fd818139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81e763700 1 --2- 172.21.15.6:0/4010374623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 0x7fd818139f40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8140099b0 tx=0x7fd814009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0dbc5cef5c55d9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 1 -- 172.21.15.6:0/4010374623 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd814009ea0 con 0x7fd818137b60 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: dump: 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:34.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: _finish_auth 0 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 1 -- 172.21.15.6:0/4010374623 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd81400ba30 con 0x7fd818137b60 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 5 monclient: authenticate success, global_id 4412 2023-09-30T00:45:34.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 1 -- 172.21.15.6:0/4010374623 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd81400bdf0 con 0x7fd818137b60 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 4 set_mon_vals no callback set 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd81cf60700 10 monclient: dump: 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.316 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 10 monclient: shutdown 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 1 -- 172.21.15.6:0/4010374623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 msgr2=0x7fd818139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.308+0000 7fd8201c6700 1 --2- 172.21.15.6:0/4010374623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 0x7fd818139f40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fd8140099b0 tx=0x7fd814009250 comp rx=0 tx=0).stop 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 1 -- 172.21.15.6:0/4010374623 shutdown_connections 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 1 --2- 172.21.15.6:0/4010374623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818137b60 0x7fd818139f40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 1 --2- 172.21.15.6:0/4010374623 >> v2:172.21.15.22:3300/0 conn(0x7fd818132d00 0x7fd818137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 1 -- 172.21.15.6:0/4010374623 >> 172.21.15.6:0/4010374623 conn(0x7fd8180a5480 msgr2=0x7fd8180a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 1 -- 172.21.15.6:0/4010374623 shutdown_connections 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 1 -- 172.21.15.6:0/4010374623 wait complete. 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 10 monclient: build_initial_monmap 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.309+0000 7fd8201c6700 10 monclient: monmap: 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:34.318 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:34.319 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:34.319 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:34.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 Processor -- start 2023-09-30T00:45:34.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 -- start start 2023-09-30T00:45:34.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient: init 2023-09-30T00:45:34.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 0x7fd8181414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): picked mon.noname-c con 0x7fd818098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd818132d00 0x7fd8181371e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): picked mon.noname-b con 0x7fd818132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd818137b60 0x7fd81809c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): picked mon.noname-a con 0x7fd818137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd81809e060 con 0x7fd818137b60 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:34.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8180a33b0 con 0x7fd818132d00 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd818133490 con 0x7fd818098670 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd8201c6700 10 monclient(hunting): authenticate will time out at 1330.608562s 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 0x7fd8181414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd818137b60 0x7fd81809c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 0x7fd8181414c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:51664/0 (socket says 172.21.15.6:51664) 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 1 -- 172.21.15.6:0/3914951538 learned_addr learned my addr 172.21.15.6:0/3914951538 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): get_auth_request con 0x7fd818098670 auth_method 0 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:34.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): get_auth_request con 0x7fd818137b60 auth_method 0 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd818132d00 0x7fd8181371e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): get_auth_request con 0x7fd818132d00 auth_method 0 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:34.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81e763700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81d761700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.310+0000 7fd81df62700 10 monclient(hunting): handle_auth_done global_id 4415 payload 824 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 1 -- 172.21.15.6:0/3914951538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd818132d00 msgr2=0x7fd8181371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd818132d00 0x7fd8181371e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 1 -- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd818137b60 msgr2=0x7fd81809c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd818137b60 0x7fd81809c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 10 monclient: _finish_hunting 0 2023-09-30T00:45:34.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 1 monclient: found mon.noname-c 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.311+0000 7fd81df62700 1 -- 172.21.15.6:0/3914951538 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8140094f0 con 0x7fd818098670 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd81df62700 1 -- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 msgr2=0x7fd8181414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd81df62700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 0x7fd8181414c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd81000c950 tx=0x7fd810002760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810012840 con 0x7fd818098670 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: dump: 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:34.327 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: _finish_auth 0 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 5 monclient: authenticate success, global_id 4415 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8100129c0 con 0x7fd818098670 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:34.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd810012d80 con 0x7fd818098670 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80dffb700 10 monclient: dump: 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:34.329 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 10 monclient: _renew_subs 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8181339d0 con 0x7fd818098670 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd80ffff700 4 set_mon_vals no callback set 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 10 monclient: _renew_subs 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.313+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd818141a00 con 0x7fd818098670 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.314+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd810011590 con 0x7fd818098670 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.314+0000 7fd8201c6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.314+0000 7fd8201c6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.314+0000 7fd8201c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:34.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.314+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd81809ed30 con 0x7fd818098670 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.314+0000 7fd80dffb700 1 --2- 172.21.15.6:0/3914951538 >> v2:172.21.15.22:6812/142603 conn(0x7fd7fc0357a0 0x7fd7fc037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.315+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd810047cd0 con 0x7fd818098670 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.315+0000 7fd81d761700 1 --2- 172.21.15.6:0/3914951538 >> v2:172.21.15.22:6812/142603 conn(0x7fd7fc0357a0 0x7fd7fc037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.315+0000 7fd81d761700 10 monclient: get_auth_request con 0x7fd7fc0357a0 auth_method 0 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.317+0000 7fd81d761700 1 --2- 172.21.15.6:0/3914951538 >> v2:172.21.15.22:6812/142603 conn(0x7fd7fc0357a0 0x7fd7fc037c50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd81809f9a0 tx=0x7fd8080091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.320+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd810011b20 con 0x7fd818098670 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.320+0000 7fd80dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:34.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.320+0000 7fd80dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.565+0000 7fd8201c6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.565+0000 7fd8201c6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.565+0000 7fd8201c6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:34.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.565+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd818133d90 con 0x7fd818098670 2023-09-30T00:45:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.565+0000 7fd80dffb700 1 -- 172.21.15.6:0/3914951538 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd810016070 con 0x7fd818098670 2023-09-30T00:45:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.566+0000 7fd80dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:34.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.566+0000 7fd80dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:34.567 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:34.567 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 >> v2:172.21.15.22:6812/142603 conn(0x7fd7fc0357a0 msgr2=0x7fd7fc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 --2- 172.21.15.6:0/3914951538 >> v2:172.21.15.22:6812/142603 conn(0x7fd7fc0357a0 0x7fd7fc037c50 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd81809f9a0 tx=0x7fd8080091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 10 monclient: shutdown 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 msgr2=0x7fd8181414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 0x7fd8181414c0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fd81000c950 tx=0x7fd810002760 comp rx=0 tx=0).stop 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 shutdown_connections 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 --2- 172.21.15.6:0/3914951538 >> v2:172.21.15.22:6812/142603 conn(0x7fd7fc0357a0 0x7fd7fc037c50 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd818137b60 0x7fd81809c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd818098670 0x7fd8181414c0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 --2- 172.21.15.6:0/3914951538 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd818132d00 0x7fd8181371e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:34.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 >> 172.21.15.6:0/3914951538 conn(0x7fd8180a5480 msgr2=0x7fd818097620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:34.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 shutdown_connections 2023-09-30T00:45:34.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:34.568+0000 7fd8201c6700 1 -- 172.21.15.6:0/3914951538 wait complete. 2023-09-30T00:45:37.583 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.722+0000 7fb452d37700 10 monclient: get_monmap_and_config 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.722+0000 7fb452d37700 10 monclient: build_initial_monmap 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.722+0000 7fb452d37700 10 monclient: monmap: 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:37.723352+0000 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:37.723352+0000 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:37.724 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:37.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.722+0000 7fb452d37700 1 Processor -- start 2023-09-30T00:45:37.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 1 -- start start 2023-09-30T00:45:37.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient: init 2023-09-30T00:45:37.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient: _add_conns ranks=[4,2,1,3,0] 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fb44c098670 0x7fb44c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): picked mon.noname-c con 0x7fb44c098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): picked mon.noname-b con 0x7fb44c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fb44c098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0a1000 con 0x7fb44c098e90 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb44c0a0a50 con 0x7fb44c098670 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c0a0730 con 0x7fb44c098a40 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:37.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb452d37700 10 monclient(hunting): authenticate will time out at 1334.021589s 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c12ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:38386/0 (socket says 172.21.15.6:38386) 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 1 -- 172.21.15.6:0/395357799 learned_addr learned my addr 172.21.15.6:0/395357799 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): get_auth_request con 0x7fb44c098a40 auth_method 0 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb44bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb450ad3700 1 --2- 172.21.15.6:0/395357799 >> v2:172.21.15.22:3300/0 conn(0x7fb44c098670 0x7fb44c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:37.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb450ad3700 10 monclient(hunting): get_auth_request con 0x7fb44c098670 auth_method 0 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb450ad3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb450ad3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.723+0000 7fb450ad3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 10 monclient(hunting): handle_auth_done global_id 4418 payload 274 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 1 -- 172.21.15.6:0/395357799 >> v2:172.21.15.22:3300/0 conn(0x7fb44c098670 msgr2=0x7fb44c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 1 --2- 172.21.15.6:0/395357799 >> v2:172.21.15.22:3300/0 conn(0x7fb44c098670 0x7fb44c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 1 -- 172.21.15.6:0/395357799 >> v1:172.21.15.22:6789/0 conn(0x7fb44c098e90 legacy=0x7fb44c12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 1 monclient: found mon.noname-b 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:37.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.724+0000 7fb44bfff700 1 -- 172.21.15.6:0/395357799 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c068df0 con 0x7fb44c098a40 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44bfff700 1 -- 172.21.15.6:0/395357799 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 msgr2=0x7fb44c12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44bfff700 1 --2- 172.21.15.6:0/395357799 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c12ede0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb4340099b0 tx=0x7fb434009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=557184a549b26695 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 1 -- 172.21.15.6:0/395357799 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43400ba30 con 0x7fb44c098a40 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: dump: 2023-09-30T00:45:37.735 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 1 -- 172.21.15.6:0/395357799 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb43400bb90 con 0x7fb44c098a40 2023-09-30T00:45:37.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb452d37700 5 monclient: authenticate success, global_id 4418 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb452d37700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb452d37700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 1 -- 172.21.15.6:0/395357799 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb434011420 con 0x7fb44c098a40 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb452d37700 4 set_mon_vals no callback set 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.726+0000 7fb44b7fe700 10 monclient: dump: 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.737 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 10 monclient: shutdown 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 -- 172.21.15.6:0/395357799 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 msgr2=0x7fb44c12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 --2- 172.21.15.6:0/395357799 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c12ede0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb4340099b0 tx=0x7fb434009250 comp rx=0 tx=0).stop 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 -- 172.21.15.6:0/395357799 shutdown_connections 2023-09-30T00:45:37.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 --2- 172.21.15.6:0/395357799 >> v2:172.21.15.22:3300/0 conn(0x7fb44c098670 0x7fb44c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 --2- 172.21.15.6:0/395357799 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c12ede0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 -- 172.21.15.6:0/395357799 >> 172.21.15.6:0/395357799 conn(0x7fb44c0a5480 msgr2=0x7fb44c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 -- 172.21.15.6:0/395357799 shutdown_connections 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 1 -- 172.21.15.6:0/395357799 wait complete. 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 10 monclient: build_initial_monmap 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.727+0000 7fb452d37700 10 monclient: monmap: 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:37.739 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 Processor -- start 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 -- start start 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient: init 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb44c098670 0x7fb44c131650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): picked mon.noname-a con 0x7fb44c098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:37.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): picked mon.noname-c con 0x7fb44c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb44c098e90 0x7fb44c141d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): picked mon.noname-b con 0x7fb44c098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1309b0 con 0x7fb44c098670 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c1306d0 con 0x7fb44c098e90 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb44c130c90 con 0x7fb44c098a40 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb452d37700 10 monclient(hunting): authenticate will time out at 1334.026851s 2023-09-30T00:45:37.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb450ad3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb44c098670 0x7fb44c131650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:38398/0 (socket says 172.21.15.6:38398) 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 1 -- 172.21.15.6:0/1255211810 learned_addr learned my addr 172.21.15.6:0/1255211810 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 10 monclient(hunting): get_auth_request con 0x7fb44c098a40 auth_method 0 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.728+0000 7fb44bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): get_auth_request con 0x7fb44c098670 auth_method 0 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:37.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb4512d4700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb44c098e90 0x7fb44c141d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb450ad3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb4512d4700 10 monclient(hunting): get_auth_request con 0x7fb44c098e90 auth_method 0 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb4512d4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb4512d4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb4512d4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 10 monclient(hunting): handle_auth_done global_id 4421 payload 824 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 1 -- 172.21.15.6:0/1255211810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb44c098e90 msgr2=0x7fb44c141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:37.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb44c098e90 0x7fb44c141d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 1 -- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb44c098670 msgr2=0x7fb44c131650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb44c098670 0x7fb44c131650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 1 monclient: found mon.noname-c 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb44bfff700 1 -- 172.21.15.6:0/1255211810 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4340094f0 con 0x7fb44c098a40 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.729+0000 7fb4512d4700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb44c098e90 0x7fb44c141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb44bfff700 1 -- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 msgr2=0x7fb44c141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb44bfff700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c141840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb434003080 tx=0x7fb434016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb434011420 con 0x7fb44c098a40 2023-09-30T00:45:37.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: dump: 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:37.745 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: _finish_auth 0 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 5 monclient: authenticate success, global_id 4421 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 10 monclient: _renew_subs 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb434020070 con 0x7fb44c098a40 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c12d550 con 0x7fb44c098a40 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb43401b450 con 0x7fb44c098a40 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:37.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb44affd700 4 set_mon_vals no callback set 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb448ff9700 10 monclient: dump: 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 10 monclient: _renew_subs 2023-09-30T00:45:37.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.731+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c131b90 con 0x7fb44c098a40 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fb434022030 con 0x7fb44c098a40 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb452d37700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb452d37700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb452d37700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c1311b0 con 0x7fb44c098a40 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb448ff9700 1 --2- 172.21.15.6:0/1255211810 >> v2:172.21.15.22:6812/142603 conn(0x7fb42c0357f0 0x7fb42c037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.732+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fb434049640 con 0x7fb44c098a40 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.733+0000 7fb450ad3700 1 --2- 172.21.15.6:0/1255211810 >> v2:172.21.15.22:6812/142603 conn(0x7fb42c0357f0 0x7fb42c037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:37.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.733+0000 7fb450ad3700 10 monclient: get_auth_request con 0x7fb42c0357f0 auth_method 0 2023-09-30T00:45:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.735+0000 7fb450ad3700 1 --2- 172.21.15.6:0/1255211810 >> v2:172.21.15.22:6812/142603 conn(0x7fb42c0357f0 0x7fb42c037ca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb43c006fd0 tx=0x7fb43c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.737+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fb44c1311b0 con 0x7fb44c098a40 2023-09-30T00:45:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.737+0000 7fb448ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:37.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.737+0000 7fb448ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.986+0000 7fb452d37700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.986+0000 7fb452d37700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:37.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.986+0000 7fb452d37700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.986+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb44c1311b0 con 0x7fb44c098a40 2023-09-30T00:45:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.987+0000 7fb448ff9700 1 -- 172.21.15.6:0/1255211810 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fb44c1311b0 con 0x7fb44c098a40 2023-09-30T00:45:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.987+0000 7fb448ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:37.988 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.987+0000 7fb448ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:37.988 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:37.988 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 >> v2:172.21.15.22:6812/142603 conn(0x7fb42c0357f0 msgr2=0x7fb42c037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:37.990 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 --2- 172.21.15.6:0/1255211810 >> v2:172.21.15.22:6812/142603 conn(0x7fb42c0357f0 0x7fb42c037ca0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb43c006fd0 tx=0x7fb43c008040 comp rx=0 tx=0).stop 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 10 monclient: shutdown 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 msgr2=0x7fb44c141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c141840 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb434003080 tx=0x7fb434016000 comp rx=0 tx=0).stop 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 shutdown_connections 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 --2- 172.21.15.6:0/1255211810 >> v2:172.21.15.22:6812/142603 conn(0x7fb42c0357f0 0x7fb42c037ca0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fb44c098e90 0x7fb44c141d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fb44c098670 0x7fb44c131650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 --2- 172.21.15.6:0/1255211810 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fb44c098a40 0x7fb44c141840 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 >> 172.21.15.6:0/1255211810 conn(0x7fb44c0a5480 msgr2=0x7fb44c0a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:37.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 shutdown_connections 2023-09-30T00:45:37.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:37.989+0000 7fb452d37700 1 -- 172.21.15.6:0/1255211810 wait complete. 2023-09-30T00:45:41.004 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:41.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.142+0000 7f31bfec9700 10 monclient: get_monmap_and_config 2023-09-30T00:45:41.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.142+0000 7f31bfec9700 10 monclient: build_initial_monmap 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.142+0000 7f31bfec9700 10 monclient: monmap: 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:41.144033+0000 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:41.144033+0000 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:41.144 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:41.145 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.143+0000 7f31bfec9700 1 Processor -- start 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.143+0000 7f31bfec9700 1 -- start start 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.143+0000 7f31bfec9700 10 monclient: init 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient: _add_conns ranks=[0,3,4,1,2] 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f31b8098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31b8132d00 0x7f31b8137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:41.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): picked mon.noname-a con 0x7f31b8132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f31b8137b60 0x7f31b8139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): picked mon.noname-c con 0x7f31b8137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31b80a1000 con 0x7f31b8098670 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b8099150 con 0x7f31b8132d00 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f31b80a0a50 con 0x7f31b8137b60 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bfec9700 10 monclient(hunting): authenticate will time out at 1337.442325s 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f31b8132d00 0x7f31b8137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bdc65700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f31b8098670 0x7f31b8098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:58172/0 (socket says 172.21.15.6:58172) 2023-09-30T00:45:41.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bdc65700 1 -- 172.21.15.6:0/2860654411 learned_addr learned my addr 172.21.15.6:0/2860654411 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): get_auth_request con 0x7f31b8132d00 auth_method 0 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31be466700 1 --2- 172.21.15.6:0/2860654411 >> v2:172.21.15.22:3300/0 conn(0x7f31b8137b60 0x7f31b8139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31be466700 10 monclient(hunting): get_auth_request con 0x7f31b8137b60 auth_method 0 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31be466700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31be466700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:41.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31be466700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient(hunting): handle_auth_done global_id 4416 payload 274 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 1 -- 172.21.15.6:0/2860654411 >> v2:172.21.15.22:3300/0 conn(0x7f31b8137b60 msgr2=0x7f31b8139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 1 --2- 172.21.15.6:0/2860654411 >> v2:172.21.15.22:3300/0 conn(0x7f31b8137b60 0x7f31b8139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 1 -- 172.21.15.6:0/2860654411 >> v1:172.21.15.6:6789/0 conn(0x7f31b8098670 legacy=0x7f31b8098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient: _finish_hunting 0 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 1 monclient: found mon.noname-a 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:41.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.144+0000 7f31bd464700 1 -- 172.21.15.6:0/2860654411 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b80a0730 con 0x7f31b8132d00 2023-09-30T00:45:41.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bd464700 1 -- 172.21.15.6:0/2860654411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8132d00 msgr2=0x7f31b8137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:41.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bd464700 1 --2- 172.21.15.6:0/2860654411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8132d00 0x7f31b8137620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31b80a3330 tx=0x7f31ac0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=941c2a2e39bc6069 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:41.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 1 -- 172.21.15.6:0/2860654411 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00b880 con 0x7f31b8132d00 2023-09-30T00:45:41.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:41.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:41.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: dump: 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:41.159 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:41.160 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:41.160 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:41.160 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:41.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: _finish_auth 0 2023-09-30T00:45:41.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:41.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 5 monclient: authenticate success, global_id 4416 2023-09-30T00:45:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 1 -- 172.21.15.6:0/2860654411 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31ac00b9e0 con 0x7f31b8132d00 2023-09-30T00:45:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 4 set_mon_vals no callback set 2023-09-30T00:45:41.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 1 -- 172.21.15.6:0/2860654411 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31ac00bd80 con 0x7f31b8132d00 2023-09-30T00:45:41.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:41.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:41.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bcc63700 10 monclient: dump: 2023-09-30T00:45:41.162 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:41.162 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:41.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.163 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.163 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:41.163 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:41.163 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:41.163 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 10 monclient: shutdown 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 -- 172.21.15.6:0/2860654411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8132d00 msgr2=0x7f31b8137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 --2- 172.21.15.6:0/2860654411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8132d00 0x7f31b8137620 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f31b80a3330 tx=0x7f31ac0091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 -- 172.21.15.6:0/2860654411 shutdown_connections 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 --2- 172.21.15.6:0/2860654411 >> v2:172.21.15.22:3300/0 conn(0x7f31b8137b60 0x7f31b8139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 --2- 172.21.15.6:0/2860654411 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8132d00 0x7f31b8137620 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 -- 172.21.15.6:0/2860654411 >> 172.21.15.6:0/2860654411 conn(0x7f31b80a5480 msgr2=0x7f31b80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 -- 172.21.15.6:0/2860654411 shutdown_connections 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.147+0000 7f31bfec9700 1 -- 172.21.15.6:0/2860654411 wait complete. 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.148+0000 7f31bfec9700 10 monclient: build_initial_monmap 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.148+0000 7f31bfec9700 10 monclient: monmap: 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:41.165 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.148+0000 7f31bfec9700 1 Processor -- start 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.148+0000 7f31bfec9700 1 -- start start 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.148+0000 7f31bfec9700 10 monclient: init 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:41.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 0x7f31b8131090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): picked mon.noname-a con 0x7f31b8098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31b8132d00 0x7f31b8134170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): picked mon.noname-b con 0x7f31b8132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f31b8137b60 0x7f31b81346b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): picked mon.noname-c con 0x7f31b8137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b812d3a0 con 0x7f31b8098670 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b81369c0 con 0x7f31b8132d00 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:41.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f31b80a1500 con 0x7f31b8137b60 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bfec9700 10 monclient(hunting): authenticate will time out at 1337.447335s 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 0x7f31b8131090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f31b8137b60 0x7f31b81346b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 0x7f31b8131090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36150/0 (socket says 172.21.15.6:36150) 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 -- 172.21.15.6:0/1887566179 learned_addr learned my addr 172.21.15.6:0/1887566179 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): get_auth_request con 0x7f31b8098670 auth_method 0 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:41.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): get_auth_request con 0x7f31b8137b60 auth_method 0 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31be466700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bd464700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31b8132d00 0x7f31b8134170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bd464700 10 monclient(hunting): get_auth_request con 0x7f31b8132d00 auth_method 0 2023-09-30T00:45:41.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bd464700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bd464700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bd464700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient(hunting): handle_auth_done global_id 4419 payload 824 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 -- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f31b8137b60 msgr2=0x7f31b81346b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f31b8137b60 0x7f31b81346b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 -- 172.21.15.6:0/1887566179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31b8132d00 msgr2=0x7f31b8134170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31b8132d00 0x7f31b8134170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient: _finish_hunting 0 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 monclient: found mon.noname-a 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:41.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.149+0000 7f31bdc65700 1 -- 172.21.15.6:0/1887566179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31ac009420 con 0x7f31b8098670 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bdc65700 1 -- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 msgr2=0x7f31b8131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bdc65700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 0x7f31b8131090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f31b400c5e0 tx=0x7f31b4004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b4005d40 con 0x7f31b8098670 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: dump: 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.171 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f31b40070d0 con 0x7f31b8098670 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31b4007490 con 0x7f31b8098670 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:41.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31a9ffb700 10 monclient: dump: 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:41.173 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 5 monclient: authenticate success, global_id 4419 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 10 monclient: _renew_subs 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31abfff700 4 set_mon_vals no callback set 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b8137700 con 0x7f31b8098670 2023-09-30T00:45:41.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 10 monclient: _renew_subs 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.152+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b8135d10 con 0x7f31b8098670 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f31b401b3b0 con 0x7f31b8098670 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31a9ffb700 1 --2- 172.21.15.6:0/1887566179 >> v2:172.21.15.22:6812/142603 conn(0x7f319c0357f0 0x7f319c037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31bfec9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f31b4046eb0 con 0x7f31b8098670 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31bfec9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31bfec9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b812d6d0 con 0x7f31b8098670 2023-09-30T00:45:41.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31bd464700 1 --2- 172.21.15.6:0/1887566179 >> v2:172.21.15.22:6812/142603 conn(0x7f319c0357f0 0x7f319c037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:41.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.153+0000 7f31bd464700 10 monclient: get_auth_request con 0x7f319c0357f0 auth_method 0 2023-09-30T00:45:41.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.156+0000 7f31bd464700 1 --2- 172.21.15.6:0/1887566179 >> v2:172.21.15.22:6812/142603 conn(0x7f319c0357f0 0x7f319c037ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0030c0 tx=0x7f31ac016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:41.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.158+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f31b4017070 con 0x7f31b8098670 2023-09-30T00:45:41.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.158+0000 7f31a9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:41.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.158+0000 7f31a9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31bfec9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:41.406 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31bfec9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31bfec9700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f31b8072b00 con 0x7f31b8098670 2023-09-30T00:45:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31a9ffb700 1 -- 172.21.15.6:0/1887566179 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f31b401a360 con 0x7f31b8098670 2023-09-30T00:45:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31a9ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:41.407 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.405+0000 7f31a9ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:41.408 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:41.408 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 >> v2:172.21.15.22:6812/142603 conn(0x7f319c0357f0 msgr2=0x7f319c037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 --2- 172.21.15.6:0/1887566179 >> v2:172.21.15.22:6812/142603 conn(0x7f319c0357f0 0x7f319c037ca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f31ac0030c0 tx=0x7f31ac016000 comp rx=0 tx=0).stop 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 10 monclient: shutdown 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 msgr2=0x7f31b8131090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 0x7f31b8131090 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f31b400c5e0 tx=0x7f31b4004a80 comp rx=0 tx=0).stop 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 shutdown_connections 2023-09-30T00:45:41.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 --2- 172.21.15.6:0/1887566179 >> v2:172.21.15.22:6812/142603 conn(0x7f319c0357f0 0x7f319c037ca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f31b8137b60 0x7f31b81346b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f31b8098670 0x7f31b8131090 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 --2- 172.21.15.6:0/1887566179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f31b8132d00 0x7f31b8134170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 >> 172.21.15.6:0/1887566179 conn(0x7f31b80a5480 msgr2=0x7f31b80a3380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 shutdown_connections 2023-09-30T00:45:41.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:41.407+0000 7f31bfec9700 1 -- 172.21.15.6:0/1887566179 wait complete. 2023-09-30T00:45:44.421 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.559+0000 7f6be51a6700 10 monclient: get_monmap_and_config 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.559+0000 7f6be51a6700 10 monclient: build_initial_monmap 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.559+0000 7f6be51a6700 10 monclient: monmap: 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:44.560368+0000 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:44.560368+0000 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:44.560 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:44.561 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:44.561 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.559+0000 7f6be51a6700 1 Processor -- start 2023-09-30T00:45:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.559+0000 7f6be51a6700 1 -- start start 2023-09-30T00:45:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.559+0000 7f6be51a6700 10 monclient: init 2023-09-30T00:45:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:44.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6be0098670 0x7f6be0132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): picked mon.noname-a con 0x7f6be0098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6be0098a40 0x7f6be012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): picked mon.noname-c con 0x7f6be0098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6be0098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6be00a1000 con 0x7f6be0098e90 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00a0a50 con 0x7f6be0098670 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6be00a0730 con 0x7f6be0098a40 2023-09-30T00:45:44.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6be51a6700 10 monclient(hunting): authenticate will time out at 1340.858462s 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6be0098670 0x7f6be0132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6be0098670 0x7f6be0132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36158/0 (socket says 172.21.15.6:36158) 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 -- 172.21.15.6:0/4034163992 learned_addr learned my addr 172.21.15.6:0/4034163992 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): get_auth_request con 0x7f6be0098670 auth_method 0 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 1 --2- 172.21.15.6:0/4034163992 >> v2:172.21.15.22:3300/0 conn(0x7f6be0098a40 0x7f6be012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): get_auth_request con 0x7f6be0098a40 auth_method 0 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:44.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient(hunting): handle_auth_done global_id 4422 payload 274 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 -- 172.21.15.6:0/4034163992 >> v2:172.21.15.22:3300/0 conn(0x7f6be0098a40 msgr2=0x7f6be012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 --2- 172.21.15.6:0/4034163992 >> v2:172.21.15.22:3300/0 conn(0x7f6be0098a40 0x7f6be012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 -- 172.21.15.6:0/4034163992 >> v1:172.21.15.22:6789/0 conn(0x7f6be0098e90 legacy=0x7f6be012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:44.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient: _finish_hunting 0 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 monclient: found mon.noname-a 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.560+0000 7f6bded9d700 1 -- 172.21.15.6:0/4034163992 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6be0068df0 con 0x7f6be0098670 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bded9d700 1 -- 172.21.15.6:0/4034163992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 msgr2=0x7f6be0132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bded9d700 1 --2- 172.21.15.6:0/4034163992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0132d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6be0063790 tx=0x7f6bd0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f03902ce9d06e20 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 1 -- 172.21.15.6:0/4034163992 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0009e70 con 0x7f6be0098670 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: dump: 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:44.569 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: _finish_auth 0 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 1 -- 172.21.15.6:0/4034163992 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd000ba30 con 0x7f6be0098670 2023-09-30T00:45:44.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 5 monclient: authenticate success, global_id 4422 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 4 set_mon_vals no callback set 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 1 -- 172.21.15.6:0/4034163992 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd000bdf0 con 0x7f6be0098670 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6bddd9b700 10 monclient: dump: 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.571 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 10 monclient: shutdown 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 1 -- 172.21.15.6:0/4034163992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 msgr2=0x7f6be0132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 1 --2- 172.21.15.6:0/4034163992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0132d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6be0063790 tx=0x7f6bd0009250 comp rx=0 tx=0).stop 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 1 -- 172.21.15.6:0/4034163992 shutdown_connections 2023-09-30T00:45:44.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 1 --2- 172.21.15.6:0/4034163992 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0132d00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 1 --2- 172.21.15.6:0/4034163992 >> v2:172.21.15.22:3300/0 conn(0x7f6be0098a40 0x7f6be012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.563+0000 7f6be51a6700 1 -- 172.21.15.6:0/4034163992 >> 172.21.15.6:0/4034163992 conn(0x7f6be00a5480 msgr2=0x7f6be00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.564+0000 7f6be51a6700 1 -- 172.21.15.6:0/4034163992 shutdown_connections 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.564+0000 7f6be51a6700 1 -- 172.21.15.6:0/4034163992 wait complete. 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.564+0000 7f6be51a6700 10 monclient: build_initial_monmap 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.564+0000 7f6be51a6700 10 monclient: monmap: 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:44.573 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:44.574 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 Processor -- start 2023-09-30T00:45:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 -- start start 2023-09-30T00:45:44.574 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient: init 2023-09-30T00:45:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:45:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): picked mon.noname-a con 0x7f6be0098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 0x7f6be012c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): picked mon.noname-c con 0x7f6be0098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6be0098e90 0x7f6be012c720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): picked mon.noname-b con 0x7f6be0098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be0142760 con 0x7f6be0098670 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6be011c610 con 0x7f6be0098e90 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:44.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6be009f2d0 con 0x7f6be0098a40 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6be51a6700 10 monclient(hunting): authenticate will time out at 1340.863591s 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 0x7f6be012c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 0x7f6be012c1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:38414/0 (socket says 172.21.15.6:38414) 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 1 -- 172.21.15.6:0/2479096752 learned_addr learned my addr 172.21.15.6:0/2479096752 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0141880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:36166/0 (socket says 172.21.15.6:36166) 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): get_auth_request con 0x7f6be0098a40 auth_method 0 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): get_auth_request con 0x7f6be0098670 auth_method 0 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bde59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bdf59e700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6be0098e90 0x7f6be012c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bdf59e700 10 monclient(hunting): get_auth_request con 0x7f6be0098e90 auth_method 0 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bdf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:44.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bdf59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.565+0000 7f6bdf59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bdf59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bdf59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bdf59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 10 monclient(hunting): handle_auth_done global_id 4425 payload 824 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 1 -- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 msgr2=0x7f6be012c1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 0x7f6be012c1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 1 -- 172.21.15.6:0/2479096752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6be0098e90 msgr2=0x7f6be012c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6be0098e90 0x7f6be012c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 10 monclient: _finish_hunting 0 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 1 monclient: found mon.noname-a 2023-09-30T00:45:44.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bded9d700 1 -- 172.21.15.6:0/2479096752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd00094f0 con 0x7f6be0098670 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.566+0000 7f6bde59c700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 0x7f6be012c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bded9d700 1 -- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 msgr2=0x7f6be0141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bded9d700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0141880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6be00a3330 tx=0x7f6bd0016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd0006b30 con 0x7f6be0098670 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: dump: 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 5 monclient: authenticate success, global_id 4425 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 10 monclient: _renew_subs 2023-09-30T00:45:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6be011c330 con 0x7f6be0098670 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bd0020070 con 0x7f6be0098670 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6bd000bdf0 con 0x7f6be0098670 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bc6ffd700 10 monclient: dump: 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 10 monclient: _renew_subs 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6be009e690 con 0x7f6be0098670 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.568+0000 7f6bdd59a700 4 set_mon_vals no callback set 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6bd0022030 con 0x7f6be0098670 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6be51a6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:44.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6be51a6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6be51a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6be00a1570 con 0x7f6be0098670 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6bc6ffd700 1 --2- 172.21.15.6:0/2479096752 >> v2:172.21.15.22:6812/142603 conn(0x7f6bcc0357a0 0x7f6bcc037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6bd00494e0 con 0x7f6be0098670 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6bde59c700 1 --2- 172.21.15.6:0/2479096752 >> v2:172.21.15.22:6812/142603 conn(0x7f6bcc0357a0 0x7f6bcc037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.569+0000 7f6bde59c700 10 monclient: get_auth_request con 0x7f6bcc0357a0 auth_method 0 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.572+0000 7f6bde59c700 1 --2- 172.21.15.6:0/2479096752 >> v2:172.21.15.22:6812/142603 conn(0x7f6bcc0357a0 0x7f6bcc037c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009fd0 tx=0x7f6bd40091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.574+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6be00a1570 con 0x7f6be0098670 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.574+0000 7f6bc6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:44.587 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.574+0000 7f6bc6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6be51a6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6be51a6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6be51a6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6be00a1570 con 0x7f6be0098670 2023-09-30T00:45:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6bc6ffd700 1 -- 172.21.15.6:0/2479096752 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6be00a1570 con 0x7f6be0098670 2023-09-30T00:45:44.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6bc6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:44.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.818+0000 7f6bc6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:44.820 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:44.820 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 >> v2:172.21.15.22:6812/142603 conn(0x7f6bcc0357a0 msgr2=0x7f6bcc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 --2- 172.21.15.6:0/2479096752 >> v2:172.21.15.22:6812/142603 conn(0x7f6bcc0357a0 0x7f6bcc037c50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4009fd0 tx=0x7f6bd40091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 10 monclient: shutdown 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 msgr2=0x7f6be0141880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0141880 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6be00a3330 tx=0x7f6bd0016000 comp rx=0 tx=0).stop 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 shutdown_connections 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 --2- 172.21.15.6:0/2479096752 >> v2:172.21.15.22:6812/142603 conn(0x7f6bcc0357a0 0x7f6bcc037c50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6be0098e90 0x7f6be012c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6be0098670 0x7f6be0141880 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 --2- 172.21.15.6:0/2479096752 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6be0098a40 0x7f6be012c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 >> 172.21.15.6:0/2479096752 conn(0x7f6be00a5480 msgr2=0x7f6be0130a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 shutdown_connections 2023-09-30T00:45:44.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:44.821+0000 7f6be51a6700 1 -- 172.21.15.6:0/2479096752 wait complete. 2023-09-30T00:45:47.836 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.973+0000 7f6d7692e700 10 monclient: get_monmap_and_config 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.973+0000 7f6d7692e700 10 monclient: build_initial_monmap 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.973+0000 7f6d7692e700 10 monclient: monmap: 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:47.974768+0000 2023-09-30T00:45:47.974 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:47.974768+0000 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:47.975 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:47.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 Processor -- start 2023-09-30T00:45:47.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 -- start start 2023-09-30T00:45:47.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient: init 2023-09-30T00:45:47.979 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient: _add_conns ranks=[0,4,2,3,1] 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6d70098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6d70132d00 0x7f6d70137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): picked mon.noname-c con 0x7f6d70132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d70139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): picked mon.noname-b con 0x7f6d70137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d700a1000 con 0x7f6d70098670 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d70099150 con 0x7f6d70132d00 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:47.980 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d700a0a50 con 0x7f6d70137b60 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.974+0000 7f6d7692e700 10 monclient(hunting): authenticate will time out at 1344.272980s 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d70139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d70139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42930/0 (socket says 172.21.15.6:42930) 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6ffff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6d70098670 0x7f6d70098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60126/0 (socket says 172.21.15.6:60126) 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 -- 172.21.15.6:0/1171905544 learned_addr learned my addr 172.21.15.6:0/1171905544 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): get_auth_request con 0x7f6d70137b60 auth_method 0 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:47.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6f7fe700 1 --2- 172.21.15.6:0/1171905544 >> v2:172.21.15.22:3300/0 conn(0x7f6d70132d00 0x7f6d70137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6f7fe700 10 monclient(hunting): get_auth_request con 0x7f6d70132d00 auth_method 0 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_done global_id 4430 payload 274 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 -- 172.21.15.6:0/1171905544 >> v2:172.21.15.22:3300/0 conn(0x7f6d70132d00 msgr2=0x7f6d70137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 --2- 172.21.15.6:0/1171905544 >> v2:172.21.15.22:3300/0 conn(0x7f6d70132d00 0x7f6d70137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 -- 172.21.15.6:0/1171905544 >> v1:172.21.15.6:6789/0 conn(0x7f6d70098670 legacy=0x7f6d70098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:47.982 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient: _finish_hunting 0 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 monclient: found mon.noname-b 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d74ecb700 1 -- 172.21.15.6:0/1171905544 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d700a0730 con 0x7f6d70137b60 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.975+0000 7f6d6f7fe700 1 --2- 172.21.15.6:0/1171905544 >> v2:172.21.15.22:3300/0 conn(0x7f6d70132d00 0x7f6d70137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.977+0000 7f6d74ecb700 1 -- 172.21.15.6:0/1171905544 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 msgr2=0x7f6d70139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.977+0000 7f6d74ecb700 1 --2- 172.21.15.6:0/1171905544 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d70139f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d68009f20 tx=0x7f6d68009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71bdbebdf97619d6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.977+0000 7f6d6effd700 1 -- 172.21.15.6:0/1171905544 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d6800e070 con 0x7f6d70137b60 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: dump: 2023-09-30T00:45:47.983 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: _finish_auth 0 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:47.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 1 -- 172.21.15.6:0/1171905544 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6800bc50 con 0x7f6d70137b60 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 1 -- 172.21.15.6:0/1171905544 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d680092e0 con 0x7f6d70137b60 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 5 monclient: authenticate success, global_id 4430 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 4 set_mon_vals no callback set 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d6effd700 10 monclient: dump: 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:47.985 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 10 monclient: shutdown 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 -- 172.21.15.6:0/1171905544 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 msgr2=0x7f6d70139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1171905544 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d70139f40 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f6d68009f20 tx=0x7f6d68009840 comp rx=0 tx=0).stop 2023-09-30T00:45:47.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 -- 172.21.15.6:0/1171905544 shutdown_connections 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1171905544 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d70139f40 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1171905544 >> v2:172.21.15.22:3300/0 conn(0x7f6d70132d00 0x7f6d70137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 -- 172.21.15.6:0/1171905544 >> 172.21.15.6:0/1171905544 conn(0x7f6d700a5480 msgr2=0x7f6d700a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 -- 172.21.15.6:0/1171905544 shutdown_connections 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.978+0000 7f6d7692e700 1 -- 172.21.15.6:0/1171905544 wait complete. 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.979+0000 7f6d7692e700 10 monclient: build_initial_monmap 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.979+0000 7f6d7692e700 10 monclient: monmap: 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:47.987 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:47.988 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:47.988 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.979+0000 7f6d7692e700 1 Processor -- start 2023-09-30T00:45:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.979+0000 7f6d7692e700 1 -- start start 2023-09-30T00:45:47.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.979+0000 7f6d7692e700 10 monclient: init 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 0x7f6d70141450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): picked mon.noname-a con 0x7f6d70098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d70132d00 0x7f6d70137200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): picked mon.noname-b con 0x7f6d70132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d7013f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): picked mon.noname-c con 0x7f6d70137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d70133410 con 0x7f6d70098670 2023-09-30T00:45:47.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d701336f0 con 0x7f6d70132d00 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d701339d0 con 0x7f6d70137b60 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d7692e700 10 monclient(hunting): authenticate will time out at 1344.278300s 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 0x7f6d70141450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d7013f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 0x7f6d70141450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44356/0 (socket says 172.21.15.6:44356) 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 -- 172.21.15.6:0/1240797112 learned_addr learned my addr 172.21.15.6:0/1240797112 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): get_auth_request con 0x7f6d70098670 auth_method 0 2023-09-30T00:45:47.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d7013f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42936/0 (socket says 172.21.15.6:42936) 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): get_auth_request con 0x7f6d70137b60 auth_method 0 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:47.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d70132d00 0x7f6d70137200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): get_auth_request con 0x7f6d70132d00 auth_method 0 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient(hunting): handle_auth_done global_id 4428 payload 824 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 -- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 msgr2=0x7f6d7013f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:47.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d7013f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 -- 172.21.15.6:0/1240797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d70132d00 msgr2=0x7f6d70137200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d70132d00 0x7f6d70137200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 monclient: found mon.noname-a 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d6ffff700 1 -- 172.21.15.6:0/1240797112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d68009ae0 con 0x7f6d70098670 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.980+0000 7f6d74ecb700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d7013f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d6ffff700 1 -- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 msgr2=0x7f6d70141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d6ffff700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 0x7f6d70141450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000c5e0 tx=0x7f6d60007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60005c80 con 0x7f6d70098670 2023-09-30T00:45:47.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: dump: 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:47.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: _finish_auth 0 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 5 monclient: authenticate success, global_id 4428 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 10 monclient: _renew_subs 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7012f240 con 0x7f6d70098670 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 10 monclient: _renew_subs 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7012e210 con 0x7f6d70098670 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d60005e00 con 0x7f6d70098670 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:47.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d60007320 con 0x7f6d70098670 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d67fff700 10 monclient: dump: 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:48.000 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.983+0000 7f6d6e7fc700 4 set_mon_vals no callback set 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6d6001b3b0 con 0x7f6d70098670 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d7692e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d7692e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d7692e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d540050b0 con 0x7f6d70098670 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d67fff700 1 --2- 172.21.15.6:0/1240797112 >> v2:172.21.15.22:6812/142603 conn(0x7f6d500357a0 0x7f6d50037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6d60046c50 con 0x7f6d70098670 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d6f7fe700 1 --2- 172.21.15.6:0/1240797112 >> v2:172.21.15.22:6812/142603 conn(0x7f6d500357a0 0x7f6d50037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.984+0000 7f6d6f7fe700 10 monclient: get_auth_request con 0x7f6d500357a0 auth_method 0 2023-09-30T00:45:48.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.987+0000 7f6d6f7fe700 1 --2- 172.21.15.6:0/1240797112 >> v2:172.21.15.22:6812/142603 conn(0x7f6d500357a0 0x7f6d50037c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d7009f830 tx=0x7f6d58008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.989+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6d6001b660 con 0x7f6d70098670 2023-09-30T00:45:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.989+0000 7f6d67fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:48.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:47.989+0000 7f6d67fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.236+0000 7f6d7692e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.236+0000 7f6d7692e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:48.237 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.236+0000 7f6d7692e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.236+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6d540058e0 con 0x7f6d70098670 2023-09-30T00:45:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.237+0000 7f6d67fff700 1 -- 172.21.15.6:0/1240797112 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6d60017070 con 0x7f6d70098670 2023-09-30T00:45:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.237+0000 7f6d67fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:48.238 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.237+0000 7f6d67fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:48.238 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:48.238 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.238+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 >> v2:172.21.15.22:6812/142603 conn(0x7f6d500357a0 msgr2=0x7f6d50037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1240797112 >> v2:172.21.15.22:6812/142603 conn(0x7f6d500357a0 0x7f6d50037c50 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f6d7009f830 tx=0x7f6d58008040 comp rx=0 tx=0).stop 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 10 monclient: shutdown 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 msgr2=0x7f6d70141450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 0x7f6d70141450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6d6000c5e0 tx=0x7f6d60007f90 comp rx=0 tx=0).stop 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 shutdown_connections 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1240797112 >> v2:172.21.15.22:6812/142603 conn(0x7f6d500357a0 0x7f6d50037c50 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6d70137b60 0x7f6d7013f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:48.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6d70098670 0x7f6d70141450 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:48.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 --2- 172.21.15.6:0/1240797112 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6d70132d00 0x7f6d70137200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:48.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 >> 172.21.15.6:0/1240797112 conn(0x7f6d700a5480 msgr2=0x7f6d70097620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:48.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 shutdown_connections 2023-09-30T00:45:48.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:48.239+0000 7f6d7692e700 1 -- 172.21.15.6:0/1240797112 wait complete. 2023-09-30T00:45:51.253 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:51.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.393+0000 7fd3d84bc700 10 monclient: get_monmap_and_config 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.393+0000 7fd3d84bc700 10 monclient: build_initial_monmap 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.393+0000 7fd3d84bc700 10 monclient: monmap: 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:51.394363+0000 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:51.394363+0000 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:51.394 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:51.395 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:51.395 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.393+0000 7fd3d84bc700 1 Processor -- start 2023-09-30T00:45:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.393+0000 7fd3d84bc700 1 -- start start 2023-09-30T00:45:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.393+0000 7fd3d84bc700 10 monclient: init 2023-09-30T00:45:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T00:45:51.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd3d0098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 0x7fd3d0137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient(hunting): picked mon.noname-b con 0x7fd3d0132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd3d0137b60 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00a1000 con 0x7fd3d0137b60 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00a0730 con 0x7fd3d0098670 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d0099150 con 0x7fd3d0132d00 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:51.398 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d84bc700 10 monclient(hunting): authenticate will time out at 1347.692653s 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 0x7fd3d0137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 0x7fd3d0137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42938/0 (socket says 172.21.15.6:42938) 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 1 -- 172.21.15.6:0/4055852077 learned_addr learned my addr 172.21.15.6:0/4055852077 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): get_auth_request con 0x7fd3d0132d00 auth_method 0 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d6a59700 1 --1- 172.21.15.6:0/4055852077 >> v1:172.21.15.6:6789/0 conn(0x7fd3d0137b60 0x7fd3d0139f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60138/0 (socket says 172.21.15.6:60138) 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:51.399 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.394+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_done global_id 4436 payload 274 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 1 -- 172.21.15.6:0/4055852077 >> v1:172.21.15.22:6789/0 conn(0x7fd3d0098670 legacy=0x7fd3d0098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 1 -- 172.21.15.6:0/4055852077 >> v1:172.21.15.6:6789/0 conn(0x7fd3d0137b60 legacy=0x7fd3d0139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 10 monclient: _finish_hunting 0 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 1 monclient: found mon.noname-b 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:51.400 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.395+0000 7fd3d5a57700 1 -- 172.21.15.6:0/4055852077 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d0068df0 con 0x7fd3d0132d00 2023-09-30T00:45:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5a57700 1 -- 172.21.15.6:0/4055852077 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 msgr2=0x7fd3d0137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:51.409 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5a57700 1 --2- 172.21.15.6:0/4055852077 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 0x7fd3d0137620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3c00099e0 tx=0x7fd3c0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c14c64b377a00b81 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 1 -- 172.21.15.6:0/4055852077 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c000bad0 con 0x7fd3d0132d00 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: dump: 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:51.410 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:51.411 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: _finish_auth 0 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 1 -- 172.21.15.6:0/4055852077 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3c000bc30 con 0x7fd3d0132d00 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d84bc700 5 monclient: authenticate success, global_id 4436 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d84bc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:51.412 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 1 -- 172.21.15.6:0/4055852077 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3c0011420 con 0x7fd3d0132d00 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d84bc700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d84bc700 4 set_mon_vals no callback set 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.397+0000 7fd3d5256700 10 monclient: dump: 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:51.413 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 10 monclient: shutdown 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 -- 172.21.15.6:0/4055852077 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 msgr2=0x7fd3d0137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/4055852077 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 0x7fd3d0137620 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd3c00099e0 tx=0x7fd3c0009250 comp rx=0 tx=0).stop 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 -- 172.21.15.6:0/4055852077 shutdown_connections 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/4055852077 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0132d00 0x7fd3d0137620 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.414 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 -- 172.21.15.6:0/4055852077 >> 172.21.15.6:0/4055852077 conn(0x7fd3d00a5480 msgr2=0x7fd3d00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 -- 172.21.15.6:0/4055852077 shutdown_connections 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 1 -- 172.21.15.6:0/4055852077 wait complete. 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 10 monclient: build_initial_monmap 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.398+0000 7fd3d84bc700 10 monclient: monmap: 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:51.415 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 Processor -- start 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 -- start start 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient: init 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 0x7fd3d012f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): picked mon.noname-c con 0x7fd3d0098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 0x7fd3d009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): picked mon.noname-a con 0x7fd3d0132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:51.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3d0137b60 0x7fd3d009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): picked mon.noname-b con 0x7fd3d0137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009dce0 con 0x7fd3d0132d00 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d00a1570 con 0x7fd3d0137b60 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3d009e930 con 0x7fd3d0098670 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d84bc700 10 monclient(hunting): authenticate will time out at 1347.697807s 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 0x7fd3d009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d6258700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 0x7fd3d012f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:51.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 0x7fd3d009c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:44368/0 (socket says 172.21.15.6:44368) 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 1 -- 172.21.15.6:0/226175499 learned_addr learned my addr 172.21.15.6:0/226175499 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 10 monclient(hunting): get_auth_request con 0x7fd3d0132d00 auth_method 0 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d5a57700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d6258700 10 monclient(hunting): get_auth_request con 0x7fd3d0098670 auth_method 0 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d6258700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d6258700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.399+0000 7fd3d6258700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3d0137b60 0x7fd3d009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:51.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d5a57700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): get_auth_request con 0x7fd3d0137b60 auth_method 0 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:51.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6a59700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 10 monclient(hunting): handle_auth_done global_id 4439 payload 824 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 1 -- 172.21.15.6:0/226175499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3d0137b60 msgr2=0x7fd3d009cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3d0137b60 0x7fd3d009cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 1 -- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 msgr2=0x7fd3d009c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 0x7fd3d009c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 10 monclient: _finish_hunting 0 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 1 monclient: found mon.noname-c 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:51.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d6258700 1 -- 172.21.15.6:0/226175499 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3c00094f0 con 0x7fd3d0098670 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.400+0000 7fd3d5a57700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 0x7fd3d009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d6258700 1 -- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 msgr2=0x7fd3d012f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d6258700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 0x7fd3d012f670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00c510 tx=0x7fd3cc007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc00ce80 con 0x7fd3d0098670 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: dump: 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:51.421 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: _finish_auth 0 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 5 monclient: authenticate success, global_id 4439 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3cc0117a0 con 0x7fd3d0098670 2023-09-30T00:45:51.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 10 monclient: _renew_subs 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd3cc011b60 con 0x7fd3d0098670 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d0133410 con 0x7fd3d0098670 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d4a55700 4 set_mon_vals no callback set 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3c67fc700 10 monclient: dump: 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.423 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 10 monclient: _renew_subs 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.402+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d0137080 con 0x7fd3d0098670 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd3cc01c460 con 0x7fd3d0098670 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3d84bc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:51.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3d84bc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3d84bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d01372a0 con 0x7fd3d0098670 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3c67fc700 1 --2- 172.21.15.6:0/226175499 >> v2:172.21.15.22:6812/142603 conn(0x7fd3b40357a0 0x7fd3b4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.403+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd3cc0474a0 con 0x7fd3d0098670 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.404+0000 7fd3d5a57700 1 --2- 172.21.15.6:0/226175499 >> v2:172.21.15.22:6812/142603 conn(0x7fd3b40357a0 0x7fd3b4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.404+0000 7fd3d5a57700 10 monclient: get_auth_request con 0x7fd3b40357a0 auth_method 0 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.406+0000 7fd3d5a57700 1 --2- 172.21.15.6:0/226175499 >> v2:172.21.15.22:6812/142603 conn(0x7fd3b40357a0 0x7fd3b4037c50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0003080 tx=0x7fd3c0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.408+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd3d01372a0 con 0x7fd3d0098670 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.408+0000 7fd3c67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:51.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.408+0000 7fd3c67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:51.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3d84bc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3d84bc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3d84bc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd3d01372a0 con 0x7fd3d0098670 2023-09-30T00:45:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3c67fc700 1 -- 172.21.15.6:0/226175499 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd3d01372a0 con 0x7fd3d0098670 2023-09-30T00:45:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3c67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:51.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.653+0000 7fd3c67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:51.655 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:51.655 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 >> v2:172.21.15.22:6812/142603 conn(0x7fd3b40357a0 msgr2=0x7fd3b4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/226175499 >> v2:172.21.15.22:6812/142603 conn(0x7fd3b40357a0 0x7fd3b4037c50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0003080 tx=0x7fd3c0016000 comp rx=0 tx=0).stop 2023-09-30T00:45:51.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 10 monclient: shutdown 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 msgr2=0x7fd3d012f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 0x7fd3d012f670 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fd3cc00c510 tx=0x7fd3cc007ea0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 shutdown_connections 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/226175499 >> v2:172.21.15.22:6812/142603 conn(0x7fd3b40357a0 0x7fd3b4037c50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd3d0137b60 0x7fd3d009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd3d0098670 0x7fd3d012f670 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 --2- 172.21.15.6:0/226175499 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd3d0132d00 0x7fd3d009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 >> 172.21.15.6:0/226175499 conn(0x7fd3d00a5480 msgr2=0x7fd3d00a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 shutdown_connections 2023-09-30T00:45:51.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:51.655+0000 7fd3d84bc700 1 -- 172.21.15.6:0/226175499 wait complete. 2023-09-30T00:45:54.671 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:54.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.809+0000 7fc90795c700 10 monclient: get_monmap_and_config 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.809+0000 7fc90795c700 10 monclient: build_initial_monmap 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.809+0000 7fc90795c700 10 monclient: monmap: 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:54.810741+0000 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:54.810741+0000 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:54.811 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 Processor -- start 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 -- start start 2023-09-30T00:45:54.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient: init 2023-09-30T00:45:54.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:54.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient: _add_conns ranks=[4,0,2,1,3] 2023-09-30T00:45:54.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc900098670 0x7fc900132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:54.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): picked mon.noname-c con 0x7fc900098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:45:54.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc900098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:54.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 0x7fc90012f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:54.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): picked mon.noname-b con 0x7fc900098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:54.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9000a1000 con 0x7fc900098a40 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9000a0a50 con 0x7fc900098670 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000a0730 con 0x7fc900098e90 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.810+0000 7fc90795c700 10 monclient(hunting): authenticate will time out at 1351.108904s 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 0x7fc90012f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc904ef7700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc900098a40 0x7fc90012ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60154/0 (socket says 172.21.15.6:60154) 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc904ef7700 1 -- 172.21.15.6:0/1594929707 learned_addr learned my addr 172.21.15.6:0/1594929707 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): get_auth_request con 0x7fc900098e90 auth_method 0 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:54.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc9056f8700 1 --2- 172.21.15.6:0/1594929707 >> v2:172.21.15.22:3300/0 conn(0x7fc900098670 0x7fc900132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient(hunting): handle_auth_done global_id 4442 payload 274 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 1 -- 172.21.15.6:0/1594929707 >> v2:172.21.15.22:3300/0 conn(0x7fc900098670 msgr2=0x7fc900132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 1 --2- 172.21.15.6:0/1594929707 >> v2:172.21.15.22:3300/0 conn(0x7fc900098670 0x7fc900132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 1 -- 172.21.15.6:0/1594929707 >> v1:172.21.15.6:6789/0 conn(0x7fc900098a40 legacy=0x7fc90012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient: _finish_hunting 0 2023-09-30T00:45:54.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 1 monclient: found mon.noname-b 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc905ef9700 1 -- 172.21.15.6:0/1594929707 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc900068df0 con 0x7fc900098e90 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc9056f8700 10 monclient: get_auth_request con 0x7fc900098670 auth_method 0 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.811+0000 7fc9056f8700 1 --2- 172.21.15.6:0/1594929707 >> v2:172.21.15.22:3300/0 conn(0x7fc900098670 0x7fc900132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc905ef9700 1 -- 172.21.15.6:0/1594929707 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 msgr2=0x7fc90012f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:54.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc905ef9700 1 --2- 172.21.15.6:0/1594929707 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 0x7fc90012f4f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009e60 tx=0x7fc8f8009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b33158800ca3575 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 1 -- 172.21.15.6:0/1594929707 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f800e070 con 0x7fc900098e90 2023-09-30T00:45:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: dump: 2023-09-30T00:45:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:54.823 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:54.824 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.824 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.824 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:54.824 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: _finish_auth 0 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:54.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc90795c700 5 monclient: authenticate success, global_id 4442 2023-09-30T00:45:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc90795c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 1 -- 172.21.15.6:0/1594929707 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8f8012430 con 0x7fc900098e90 2023-09-30T00:45:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc90795c700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc90795c700 4 set_mon_vals no callback set 2023-09-30T00:45:54.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 1 -- 172.21.15.6:0/1594929707 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8f80127d0 con 0x7fc900098e90 2023-09-30T00:45:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.813+0000 7fc8f7fff700 10 monclient: dump: 2023-09-30T00:45:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:54.827 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:54.828 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:54.829 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 10 monclient: shutdown 2023-09-30T00:45:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 -- 172.21.15.6:0/1594929707 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 msgr2=0x7fc90012f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 --2- 172.21.15.6:0/1594929707 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 0x7fc90012f4f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fc8f8009e60 tx=0x7fc8f8009780 comp rx=0 tx=0).stop 2023-09-30T00:45:54.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 -- 172.21.15.6:0/1594929707 shutdown_connections 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 --2- 172.21.15.6:0/1594929707 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098e90 0x7fc90012f4f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 --2- 172.21.15.6:0/1594929707 >> v2:172.21.15.22:3300/0 conn(0x7fc900098670 0x7fc900132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 -- 172.21.15.6:0/1594929707 >> 172.21.15.6:0/1594929707 conn(0x7fc9000a5480 msgr2=0x7fc9000a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 -- 172.21.15.6:0/1594929707 shutdown_connections 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.814+0000 7fc90795c700 1 -- 172.21.15.6:0/1594929707 wait complete. 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.815+0000 7fc90795c700 10 monclient: build_initial_monmap 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.815+0000 7fc90795c700 10 monclient: monmap: 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:54.830 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.815+0000 7fc90795c700 1 Processor -- start 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.815+0000 7fc90795c700 1 -- start start 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.815+0000 7fc90795c700 10 monclient: init 2023-09-30T00:45:54.831 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 0x7fc900141a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): picked mon.noname-c con 0x7fc900098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc900098a40 0x7fc900142270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): picked mon.noname-a con 0x7fc900098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc900098e90 0x7fc90009ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): picked mon.noname-b con 0x7fc900098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90009d3a0 con 0x7fc900098a40 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc90012ef60 con 0x7fc900098e90 2023-09-30T00:45:54.832 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9000a1530 con 0x7fc900098670 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc90795c700 10 monclient(hunting): authenticate will time out at 1351.114116s 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 0x7fc900141a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc900098a40 0x7fc900142270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 0x7fc900141a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42970/0 (socket says 172.21.15.6:42970) 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 -- 172.21.15.6:0/3869948310 learned_addr learned my addr 172.21.15.6:0/3869948310 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): get_auth_request con 0x7fc900098670 auth_method 0 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:54.833 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): get_auth_request con 0x7fc900098a40 auth_method 0 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc905ef9700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc900098e90 0x7fc90009ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:54.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc905ef9700 10 monclient(hunting): get_auth_request con 0x7fc900098e90 auth_method 0 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc905ef9700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc905ef9700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc905ef9700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient(hunting): handle_auth_done global_id 4445 payload 824 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 -- 172.21.15.6:0/3869948310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc900098e90 msgr2=0x7fc90009ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc900098e90 0x7fc90009ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 -- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc900098a40 msgr2=0x7fc900142270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc900098a40 0x7fc900142270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient: _finish_hunting 0 2023-09-30T00:45:54.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 monclient: found mon.noname-c 2023-09-30T00:45:54.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:54.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:45:54.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc9056f8700 1 -- 172.21.15.6:0/3869948310 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8f8009a20 con 0x7fc900098670 2023-09-30T00:45:54.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.816+0000 7fc904ef7700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc900098a40 0x7fc900142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:54.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.818+0000 7fc9056f8700 1 -- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 msgr2=0x7fc900141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:54.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.818+0000 7fc9056f8700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 0x7fc900141a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00c830 tx=0x7fc8fc008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc011810 con 0x7fc900098670 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: dump: 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:54.839 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: _finish_auth 0 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8fc011970 con 0x7fc900098670 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 5 monclient: authenticate success, global_id 4445 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 10 monclient: _renew_subs 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc90009cde0 con 0x7fc900098670 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:54.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8fc011d30 con 0x7fc900098670 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 10 monclient: dump: 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:54.841 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f77fe700 4 set_mon_vals no callback set 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 10 monclient: _renew_subs 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc90012ea80 con 0x7fc900098670 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.819+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc8fc0195c0 con 0x7fc900098670 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc8f57fa700 1 --2- 172.21.15.6:0/3869948310 >> v2:172.21.15.22:6812/142603 conn(0x7fc8e40357f0 0x7fc8e4037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc8fc0479e0 con 0x7fc900098670 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc90795c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc90795c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:54.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc90795c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc900064560 con 0x7fc900098670 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc904ef7700 1 --2- 172.21.15.6:0/3869948310 >> v2:172.21.15.22:6812/142603 conn(0x7fc8e40357f0 0x7fc8e4037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.820+0000 7fc904ef7700 10 monclient: get_auth_request con 0x7fc8e40357f0 auth_method 0 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.823+0000 7fc904ef7700 1 --2- 172.21.15.6:0/3869948310 >> v2:172.21.15.22:6812/142603 conn(0x7fc8e40357f0 0x7fc8e4037ca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc90012e570 tx=0x7fc8f0007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.825+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc8fc018360 con 0x7fc900098670 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.825+0000 7fc8f57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:54.843 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:54.825+0000 7fc8f57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:55.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.070+0000 7fc90795c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.070+0000 7fc90795c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.070+0000 7fc90795c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:45:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.070+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc900072b00 con 0x7fc900098670 2023-09-30T00:45:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.071+0000 7fc8f57fa700 1 -- 172.21.15.6:0/3869948310 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc8fc028080 con 0x7fc900098670 2023-09-30T00:45:55.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.071+0000 7fc8f57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:55.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.071+0000 7fc8f57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:55.076 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:55.076 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 >> v2:172.21.15.22:6812/142603 conn(0x7fc8e40357f0 msgr2=0x7fc8e4037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 --2- 172.21.15.6:0/3869948310 >> v2:172.21.15.22:6812/142603 conn(0x7fc8e40357f0 0x7fc8e4037ca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc90012e570 tx=0x7fc8f0007f90 comp rx=0 tx=0).stop 2023-09-30T00:45:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 10 monclient: shutdown 2023-09-30T00:45:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:55.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 msgr2=0x7fc900141a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 0x7fc900141a20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc8fc00c830 tx=0x7fc8fc008040 comp rx=0 tx=0).stop 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 shutdown_connections 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 --2- 172.21.15.6:0/3869948310 >> v2:172.21.15.22:6812/142603 conn(0x7fc8e40357f0 0x7fc8e4037ca0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc900098e90 0x7fc90009ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc900098670 0x7fc900141a20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 --2- 172.21.15.6:0/3869948310 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc900098a40 0x7fc900142270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 >> 172.21.15.6:0/3869948310 conn(0x7fc9000a5480 msgr2=0x7fc900069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 shutdown_connections 2023-09-30T00:45:55.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:55.073+0000 7fc90795c700 1 -- 172.21.15.6:0/3869948310 wait complete. 2023-09-30T00:45:58.087 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.226+0000 7fd9eca1e700 10 monclient: get_monmap_and_config 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.226+0000 7fd9eca1e700 10 monclient: build_initial_monmap 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.226+0000 7fd9eca1e700 10 monclient: monmap: 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:45:58.227408+0000 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:45:58.227408+0000 2023-09-30T00:45:58.227 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:45:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:45:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:45:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:45:58.228 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:45:58.229 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:58.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.226+0000 7fd9eca1e700 1 Processor -- start 2023-09-30T00:45:58.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.226+0000 7fd9eca1e700 1 -- start start 2023-09-30T00:45:58.240 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.226+0000 7fd9eca1e700 10 monclient: init 2023-09-30T00:45:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-09-30T00:45:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd9e8098570 addr v1:172.21.15.6:6789/0 2023-09-30T00:45:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e81374a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:58.241 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): picked mon.noname-b con 0x7fd9e8132ca0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 0x7fd9e8139dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): picked mon.noname-a con 0x7fd9e81379e0 addr v2:172.21.15.6:3300/0 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9e80a08c0 con 0x7fd9e8098570 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd9e80a1bf0 con 0x7fd9e81379e0 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e8096a80 con 0x7fd9e8132ca0 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:58.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9eca1e700 10 monclient(hunting): authenticate will time out at 1354.525629s 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 0x7fd9e8139dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e81374a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e659c700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd9e8098570 0x7fd9e8098940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60500/0 (socket says 172.21.15.6:60500) 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 0x7fd9e8139dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49496/0 (socket says 172.21.15.6:49496) 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e81374a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34294/0 (socket says 172.21.15.6:34294) 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e659c700 1 -- 172.21.15.6:0/3633288966 learned_addr learned my addr 172.21.15.6:0/3633288966 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): get_auth_request con 0x7fd9e81379e0 auth_method 0 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:58.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): get_auth_request con 0x7fd9e8132ca0 auth_method 0 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e6d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.227+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_done global_id 4448 payload 274 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 1 -- 172.21.15.6:0/3633288966 >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 msgr2=0x7fd9e8139dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:58.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 1 --2- 172.21.15.6:0/3633288966 >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 0x7fd9e8139dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 1 -- 172.21.15.6:0/3633288966 >> v1:172.21.15.6:6789/0 conn(0x7fd9e8098570 legacy=0x7fd9e8098940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 1 monclient: found mon.noname-b 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e5d9b700 1 -- 172.21.15.6:0/3633288966 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8096d60 con 0x7fd9e8132ca0 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.228+0000 7fd9e6d9d700 1 --2- 172.21.15.6:0/3633288966 >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 0x7fd9e8139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e5d9b700 1 -- 172.21.15.6:0/3633288966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 msgr2=0x7fd9e81374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e5d9b700 1 --2- 172.21.15.6:0/3633288966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e81374a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098e0 tx=0x7fd9d00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77e1a5fa9f2aed1f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 1 -- 172.21.15.6:0/3633288966 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0011840 con 0x7fd9e8132ca0 2023-09-30T00:45:58.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: dump: 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:58.246 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: _finish_auth 0 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 1 -- 172.21.15.6:0/3633288966 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d00119a0 con 0x7fd9e8132ca0 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9eca1e700 5 monclient: authenticate success, global_id 4448 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9eca1e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9eca1e700 10 monclient: get_monmap_and_config success 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 1 -- 172.21.15.6:0/3633288966 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d0011d40 con 0x7fd9e8132ca0 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9eca1e700 4 set_mon_vals no callback set 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:58.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.230+0000 7fd9e559a700 10 monclient: dump: 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:58.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 10 monclient: shutdown 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 -- 172.21.15.6:0/3633288966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 msgr2=0x7fd9e81374a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/3633288966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e81374a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fd9d00098e0 tx=0x7fd9d00091a0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 -- 172.21.15.6:0/3633288966 shutdown_connections 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/3633288966 >> v2:172.21.15.6:3300/0 conn(0x7fd9e81379e0 0x7fd9e8139dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/3633288966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e81374a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 -- 172.21.15.6:0/3633288966 >> 172.21.15.6:0/3633288966 conn(0x7fd9e80a1eb0 msgr2=0x7fd9e80a22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 -- 172.21.15.6:0/3633288966 shutdown_connections 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 1 -- 172.21.15.6:0/3633288966 wait complete. 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 10 monclient: build_initial_monmap 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.231+0000 7fd9eca1e700 10 monclient: monmap: 2023-09-30T00:45:58.249 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 Processor -- start 2023-09-30T00:45:58.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 -- start start 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient: init 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient: _reopen_session rank -1 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 0x7fd9e81423d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): picked mon.noname-a con 0x7fd9e8098570 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e8142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): picked mon.noname-c con 0x7fd9e8132ca0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9e81379e0 0x7fd9e81331a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): picked mon.noname-b con 0x7fd9e81379e0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e8136700 con 0x7fd9e8098570 2023-09-30T00:45:58.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80a1170 con 0x7fd9e81379e0 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9e80a3630 con 0x7fd9e8132ca0 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): _renew_subs 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9eca1e700 10 monclient(hunting): authenticate will time out at 1354.530664s 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e8142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 0x7fd9e81423d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e8142910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34310/0 (socket says 172.21.15.6:34310) 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 1 -- 172.21.15.6:0/2729190755 learned_addr learned my addr 172.21.15.6:0/2729190755 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): get_auth_request con 0x7fd9e8132ca0 auth_method 0 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:58.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): get_auth_request con 0x7fd9e8098570 auth_method 0 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e5d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.232+0000 7fd9e659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:45:58.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e6d9d700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9e81379e0 0x7fd9e81331a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 10 monclient(hunting): handle_auth_done global_id 4440 payload 824 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 1 -- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 msgr2=0x7fd9e8142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e8142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 1 -- 172.21.15.6:0/2729190755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9e81379e0 msgr2=0x7fd9e81331a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9e81379e0 0x7fd9e81331a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 10 monclient: _finish_hunting 0 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 1 monclient: found mon.noname-a 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e659c700 1 -- 172.21.15.6:0/2729190755 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e0009a20 con 0x7fd9e8098570 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e6d9d700 10 monclient: get_auth_request con 0x7fd9e81379e0 auth_method 0 2023-09-30T00:45:58.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e5d9b700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e8142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.233+0000 7fd9e6d9d700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9e81379e0 0x7fd9e81331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9e659c700 1 -- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 msgr2=0x7fd9e81423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9e659c700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 0x7fd9e81423d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800b4e0 tx=0x7fd9d8009060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8011840 con 0x7fd9e8098570 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: dump: 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.255 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: _finish_auth 0 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd9d80119a0 con 0x7fd9e8098570 2023-09-30T00:45:58.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 5 monclient: authenticate success, global_id 4440 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 10 monclient: _renew_subs 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd9d8011d60 con 0x7fd9e8098570 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9d0009420 con 0x7fd9e8098570 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:45:58.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:45:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9e4d99700 4 set_mon_vals no callback set 2023-09-30T00:45:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9de7fc700 10 monclient: dump: 2023-09-30T00:45:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:45:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:45:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.258 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:45:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:45:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:45:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:45:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:45:58.259 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:45:58.259 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 10 monclient: _renew_subs 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.235+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9e8136050 con 0x7fd9e8098570 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9eca1e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd9d8010590 con 0x7fd9e8098570 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9eca1e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:58.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9eca1e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9e8097ca0 con 0x7fd9e8098570 2023-09-30T00:45:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9de7fc700 1 --2- 172.21.15.6:0/2729190755 >> v2:172.21.15.22:6812/142603 conn(0x7fd9c80357a0 0x7fd9c8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:45:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.236+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd9d80478e0 con 0x7fd9e8098570 2023-09-30T00:45:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.237+0000 7fd9e5d9b700 1 --2- 172.21.15.6:0/2729190755 >> v2:172.21.15.22:6812/142603 conn(0x7fd9c80357a0 0x7fd9c8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:45:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.237+0000 7fd9e5d9b700 10 monclient: get_auth_request con 0x7fd9c80357a0 auth_method 0 2023-09-30T00:45:58.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.239+0000 7fd9e5d9b700 1 --2- 172.21.15.6:0/2729190755 >> v2:172.21.15.22:6812/142603 conn(0x7fd9c80357a0 0x7fd9c8037c50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8069dd0 tx=0x7fd9d0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:45:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.241+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd9d800f8b0 con 0x7fd9e8098570 2023-09-30T00:45:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.241+0000 7fd9de7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:45:58.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.241+0000 7fd9de7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:45:58.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9eca1e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:45:58.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9eca1e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:58.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9eca1e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:45:58.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd9e8072b00 con 0x7fd9e8098570 2023-09-30T00:45:58.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9de7fc700 1 -- 172.21.15.6:0/2729190755 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd9d8015070 con 0x7fd9e8098570 2023-09-30T00:45:58.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9de7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:45:58.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.487+0000 7fd9de7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:45:58.489 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:45:58.489 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:45:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 >> v2:172.21.15.22:6812/142603 conn(0x7fd9c80357a0 msgr2=0x7fd9c8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:58.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/2729190755 >> v2:172.21.15.22:6812/142603 conn(0x7fd9c80357a0 0x7fd9c8037c50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8069dd0 tx=0x7fd9d0016000 comp rx=0 tx=0).stop 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 10 monclient: shutdown 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 msgr2=0x7fd9e81423d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 0x7fd9e81423d0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fd9d800b4e0 tx=0x7fd9d8009060 comp rx=0 tx=0).stop 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 shutdown_connections 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/2729190755 >> v2:172.21.15.22:6812/142603 conn(0x7fd9c80357a0 0x7fd9c8037c50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd9e81379e0 0x7fd9e81331a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd9e8098570 0x7fd9e81423d0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 --2- 172.21.15.6:0/2729190755 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd9e8132ca0 0x7fd9e8142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:45:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 >> 172.21.15.6:0/2729190755 conn(0x7fd9e80a1eb0 msgr2=0x7fd9e8068d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:45:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 shutdown_connections 2023-09-30T00:45:58.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:45:58.489+0000 7fd9eca1e700 1 -- 172.21.15.6:0/2729190755 wait complete. 2023-09-30T00:46:01.504 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:01.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.642+0000 7f44a59c5700 10 monclient: get_monmap_and_config 2023-09-30T00:46:01.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.642+0000 7f44a59c5700 10 monclient: build_initial_monmap 2023-09-30T00:46:01.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.642+0000 7f44a59c5700 10 monclient: monmap: 2023-09-30T00:46:01.644 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:01.644 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:01.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:01.643881+0000 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:01.643881+0000 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 1 Processor -- start 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 1 -- start start 2023-09-30T00:46:01.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient: init 2023-09-30T00:46:01.651 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient: _add_conns ranks=[3,0,1,4,2] 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44a0132150 0x7f44a0132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient(hunting): picked mon.noname-a con 0x7f44a0132150 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f44a0132e10 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f44a0098e70 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a0132520 con 0x7f44a0132e10 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44a0096a80 con 0x7f44a0098e70 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44a00a0f80 con 0x7f44a0132150 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:01.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f44a59c5700 10 monclient(hunting): authenticate will time out at 1357.941853s 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f449effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f44a0132150 0x7f44a0132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f449e7fc700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f44a0132e10 0x7f44a0098760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:60514/0 (socket says 172.21.15.6:60514) 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.643+0000 7f449e7fc700 1 -- 172.21.15.6:0/2436717734 learned_addr learned my addr 172.21.15.6:0/2436717734 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): get_auth_request con 0x7f44a0132150 auth_method 0 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient(hunting): handle_auth_done global_id 4443 payload 274 2023-09-30T00:46:01.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 1 -- 172.21.15.6:0/2436717734 >> v1:172.21.15.22:6789/0 conn(0x7f44a0098e70 legacy=0x7f44a01394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 1 -- 172.21.15.6:0/2436717734 >> v1:172.21.15.6:6789/0 conn(0x7f44a0132e10 legacy=0x7f44a0098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient: _finish_hunting 0 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 1 monclient: found mon.noname-a 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.644+0000 7f449effd700 1 -- 172.21.15.6:0/2436717734 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44a0096d20 con 0x7f44a0132150 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449effd700 1 -- 172.21.15.6:0/2436717734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0132150 msgr2=0x7f44a0132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449effd700 1 --2- 172.21.15.6:0/2436717734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0132150 0x7f44a0132790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f44900098e0 tx=0x7f44900091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67d6001485e8362e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 1 -- 172.21.15.6:0/2436717734 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449000ba30 con 0x7f44a0132150 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:01.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: dump: 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:01.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f44a59c5700 5 monclient: authenticate success, global_id 4443 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f44a59c5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 1 -- 172.21.15.6:0/2436717734 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449000bb90 con 0x7f44a0132150 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f44a59c5700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f44a59c5700 4 set_mon_vals no callback set 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 1 -- 172.21.15.6:0/2436717734 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490011420 con 0x7f44a0132150 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.646+0000 7f449dffb700 10 monclient: dump: 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:01.656 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 10 monclient: shutdown 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:01.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 -- 172.21.15.6:0/2436717734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0132150 msgr2=0x7f44a0132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 --2- 172.21.15.6:0/2436717734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0132150 0x7f44a0132790 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f44900098e0 tx=0x7f44900091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 -- 172.21.15.6:0/2436717734 shutdown_connections 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 --2- 172.21.15.6:0/2436717734 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0132150 0x7f44a0132790 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 -- 172.21.15.6:0/2436717734 >> 172.21.15.6:0/2436717734 conn(0x7f44a00a54c0 msgr2=0x7f44a00a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 -- 172.21.15.6:0/2436717734 shutdown_connections 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.647+0000 7f44a59c5700 1 -- 172.21.15.6:0/2436717734 wait complete. 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 10 monclient: build_initial_monmap 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 10 monclient: monmap: 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:01.658 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 1 Processor -- start 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 1 -- start start 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 10 monclient: init 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:01.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 0x7f44a0133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 10 monclient(hunting): picked mon.noname-a con 0x7f44a0098e70 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.648+0000 7f44a59c5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f44a0132150 0x7f44a0133750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): picked mon.noname-c con 0x7f44a0132150 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44a0132e10 0x7f44a0133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): picked mon.noname-b con 0x7f44a0132e10 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0130060 con 0x7f44a0098e70 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44a01365d0 con 0x7f44a0132e10 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:01.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44a0135400 con 0x7f44a0132150 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f44a59c5700 10 monclient(hunting): authenticate will time out at 1357.947056s 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 0x7f44a0133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 0x7f44a0133210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49528/0 (socket says 172.21.15.6:49528) 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 -- 172.21.15.6:0/4169056155 learned_addr learned my addr 172.21.15.6:0/4169056155 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): get_auth_request con 0x7f44a0098e70 auth_method 0 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f44a0132150 0x7f44a0133750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): get_auth_request con 0x7f44a0132150 auth_method 0 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:01.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449f7fe700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44a0132e10 0x7f44a0133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449f7fe700 10 monclient(hunting): get_auth_request con 0x7f44a0132e10 auth_method 0 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:01.662 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient(hunting): handle_auth_done global_id 4446 payload 824 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 -- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f44a0132150 msgr2=0x7f44a0133750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f44a0132150 0x7f44a0133750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 -- 172.21.15.6:0/4169056155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44a0132e10 msgr2=0x7f44a0133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44a0132e10 0x7f44a0133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient: _finish_hunting 0 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 monclient: found mon.noname-a 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:01.663 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.649+0000 7f449effd700 1 -- 172.21.15.6:0/4169056155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4490009420 con 0x7f44a0098e70 2023-09-30T00:46:01.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f449effd700 1 -- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 msgr2=0x7f44a0133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:01.665 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f449effd700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 0x7f44a0133210 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f44a006a680 tx=0x7f4490016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4490011420 con 0x7f44a0098e70 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 10 monclient: dump: 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:01.666 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f449001b680 con 0x7f44a0098e70 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f44a59c5700 5 monclient: authenticate success, global_id 4446 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f44a59c5700 10 monclient: _renew_subs 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f44a59c5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44a01362f0 con 0x7f44a0098e70 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f449001ba00 con 0x7f44a0098e70 2023-09-30T00:46:01.667 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.651+0000 7f4486ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f4486ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f4486ffd700 10 monclient: dump: 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:01.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f449d7fa700 4 set_mon_vals no callback set 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f44a59c5700 10 monclient: _renew_subs 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f44a59c5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44a0137510 con 0x7f44a0098e70 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.652+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f449001c450 con 0x7f44a0098e70 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f4486ffd700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4490049500 con 0x7f44a0098e70 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f44a59c5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f44a59c5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f44a59c5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44a00a4420 con 0x7f44a0098e70 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f449e7fc700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:01.669 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f449e7fc700 10 monclient: get_auth_request con 0x7f448c0357a0 auth_method 0 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f449e7fc700 1 -- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 msgr2=0x7f448c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f449e7fc700 1 -- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 msgr2=0x7f448c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f449e7fc700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 secure :-1 s=AUTH_CONNECTING_SIGN pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f44a01331e0 tx=0x7f4494007f90 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.653+0000 7f449e7fc700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.658+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4490018760 con 0x7f44a0098e70 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.658+0000 7f4486ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:01.670 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.658+0000 7f4486ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.854+0000 7f449e7fc700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:01.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.854+0000 7f449e7fc700 10 monclient: get_auth_request con 0x7f448c0357a0 auth_method 0 2023-09-30T00:46:01.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.856+0000 7f449e7fc700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44a01331e0 tx=0x7f4494007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.907+0000 7f44a59c5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.907+0000 7f44a59c5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.907+0000 7f44a59c5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.907+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f44a00a4420 con 0x7f44a0098e70 2023-09-30T00:46:01.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.908+0000 7f4486ffd700 1 -- 172.21.15.6:0/4169056155 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4490033080 con 0x7f44a0098e70 2023-09-30T00:46:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.908+0000 7f4486ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:01.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.908+0000 7f4486ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:01.910 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:01.910 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 msgr2=0x7f448c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f44a01331e0 tx=0x7f4494007f90 comp rx=0 tx=0).stop 2023-09-30T00:46:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 10 monclient: shutdown 2023-09-30T00:46:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:01.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 msgr2=0x7f44a0133210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 0x7f44a0133210 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f44a006a680 tx=0x7f4490016000 comp rx=0 tx=0).stop 2023-09-30T00:46:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 shutdown_connections 2023-09-30T00:46:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 --2- 172.21.15.6:0/4169056155 >> v2:172.21.15.22:6812/142603 conn(0x7f448c0357a0 0x7f448c037c50 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f44a0132e10 0x7f44a0133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f44a0098e70 0x7f44a0133210 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 --2- 172.21.15.6:0/4169056155 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f44a0132150 0x7f44a0133750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 >> 172.21.15.6:0/4169056155 conn(0x7f44a00a54c0 msgr2=0x7f44a00974e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 shutdown_connections 2023-09-30T00:46:01.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:01.910+0000 7f44a59c5700 1 -- 172.21.15.6:0/4169056155 wait complete. 2023-09-30T00:46:04.925 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:05.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.063+0000 7f9ce3153700 10 monclient: get_monmap_and_config 2023-09-30T00:46:05.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.063+0000 7f9ce3153700 10 monclient: build_initial_monmap 2023-09-30T00:46:05.064 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.063+0000 7f9ce3153700 10 monclient: monmap: 2023-09-30T00:46:05.064 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:05.064 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:05.064833+0000 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:05.064833+0000 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:05.065 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 1 Processor -- start 2023-09-30T00:46:05.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 1 -- start start 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 10 monclient: init 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9cdc098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.064+0000 7f9ce3153700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9cdc132d00 0x7f9cdc137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 10 monclient(hunting): picked mon.noname-c con 0x7f9cdc132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cdc137b60 0x7f9cdc139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 10 monclient(hunting): picked mon.noname-a con 0x7f9cdc137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cdc0a1000 con 0x7f9cdc098670 2023-09-30T00:46:05.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc099150 con 0x7f9cdc137b60 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cdc0a0a50 con 0x7f9cdc132d00 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce3153700 10 monclient(hunting): authenticate will time out at 1361.363073s 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cdc137b60 0x7f9cdc139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9cdc137b60 0x7f9cdc139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49546/0 (socket says 172.21.15.6:49546) 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 -- 172.21.15.6:0/2755212993 learned_addr learned my addr 172.21.15.6:0/2755212993 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): get_auth_request con 0x7f9cdc137b60 auth_method 0 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:05.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 1 --2- 172.21.15.6:0/2755212993 >> v2:172.21.15.22:3300/0 conn(0x7f9cdc132d00 0x7f9cdc137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): get_auth_request con 0x7f9cdc132d00 auth_method 0 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9cdbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:05.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_done global_id 4449 payload 274 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 -- 172.21.15.6:0/2755212993 >> v2:172.21.15.22:3300/0 conn(0x7f9cdc132d00 msgr2=0x7f9cdc137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 --2- 172.21.15.6:0/2755212993 >> v2:172.21.15.22:3300/0 conn(0x7f9cdc132d00 0x7f9cdc137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 -- 172.21.15.6:0/2755212993 >> v1:172.21.15.22:6789/0 conn(0x7f9cdc098670 legacy=0x7f9cdc098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient: _finish_hunting 0 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 monclient: found mon.noname-a 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.065+0000 7f9ce16f0700 1 -- 172.21.15.6:0/2755212993 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cdc0a0730 con 0x7f9cdc137b60 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.067+0000 7f9ce16f0700 1 -- 172.21.15.6:0/2755212993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc137b60 msgr2=0x7f9cdc139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.067+0000 7f9ce16f0700 1 --2- 172.21.15.6:0/2755212993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc137b60 0x7f9cdc139f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc063790 tx=0x7f9cd4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddbbd62f36fbe14 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 1 -- 172.21.15.6:0/2755212993 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd400e070 con 0x7f9cdc137b60 2023-09-30T00:46:05.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: dump: 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:05.074 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 5 monclient: authenticate success, global_id 4449 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 1 -- 172.21.15.6:0/2755212993 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd400bbf0 con 0x7f9cdc137b60 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 4 set_mon_vals no callback set 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 1 -- 172.21.15.6:0/2755212993 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd40092e0 con 0x7f9cdc137b60 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:05.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9cdb7fe700 10 monclient: dump: 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 10 monclient: shutdown 2023-09-30T00:46:05.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 -- 172.21.15.6:0/2755212993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc137b60 msgr2=0x7f9cdc139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 --2- 172.21.15.6:0/2755212993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc137b60 0x7f9cdc139f40 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc063790 tx=0x7f9cd4009840 comp rx=0 tx=0).stop 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 -- 172.21.15.6:0/2755212993 shutdown_connections 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 --2- 172.21.15.6:0/2755212993 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc137b60 0x7f9cdc139f40 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 --2- 172.21.15.6:0/2755212993 >> v2:172.21.15.22:3300/0 conn(0x7f9cdc132d00 0x7f9cdc137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 -- 172.21.15.6:0/2755212993 >> 172.21.15.6:0/2755212993 conn(0x7f9cdc0a5480 msgr2=0x7f9cdc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 -- 172.21.15.6:0/2755212993 shutdown_connections 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.068+0000 7f9ce3153700 1 -- 172.21.15.6:0/2755212993 wait complete. 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.069+0000 7f9ce3153700 10 monclient: build_initial_monmap 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.069+0000 7f9ce3153700 10 monclient: monmap: 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:05.077 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:05.078 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.069+0000 7f9ce3153700 1 Processor -- start 2023-09-30T00:46:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.069+0000 7f9ce3153700 1 -- start start 2023-09-30T00:46:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.069+0000 7f9ce3153700 10 monclient: init 2023-09-30T00:46:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:46:05.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 0x7f9cdc12f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): picked mon.noname-a con 0x7f9cdc098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9cdc132d00 0x7f9cdc09ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): picked mon.noname-c con 0x7f9cdc132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cdc137b60 0x7f9cdc1369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): picked mon.noname-b con 0x7f9cdc137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc0a1500 con 0x7f9cdc098670 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc137570 con 0x7f9cdc137b60 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cdc09dc80 con 0x7f9cdc132d00 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:05.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce3153700 10 monclient(hunting): authenticate will time out at 1361.368387s 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 0x7f9cdc12f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 0x7f9cdc12f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49562/0 (socket says 172.21.15.6:49562) 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9cdc132d00 0x7f9cdc09ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 -- 172.21.15.6:0/399917343 learned_addr learned my addr 172.21.15.6:0/399917343 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): get_auth_request con 0x7f9cdc098670 auth_method 0 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): get_auth_request con 0x7f9cdc132d00 auth_method 0 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:05.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cdc137b60 0x7f9cdc1369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): get_auth_request con 0x7f9cdc137b60 auth_method 0 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:05.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce16f0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient(hunting): handle_auth_done global_id 4452 payload 824 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 -- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9cdc132d00 msgr2=0x7f9cdc09ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9cdc132d00 0x7f9cdc09ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 -- 172.21.15.6:0/399917343 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cdc137b60 msgr2=0x7f9cdc1369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cdc137b60 0x7f9cdc1369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient: _finish_hunting 0 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 monclient: found mon.noname-a 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:05.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9ce0eef700 1 -- 172.21.15.6:0/399917343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4009ae0 con 0x7f9cdc098670 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.070+0000 7f9cdbfff700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9cdc132d00 0x7f9cdc09ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce0eef700 1 -- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 msgr2=0x7f9cdc12f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce0eef700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 0x7f9cdc12f110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000c600 tx=0x7f9cd0007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd00117a0 con 0x7f9cdc098670 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: dump: 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.088 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 5 monclient: authenticate success, global_id 4452 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cd0011900 con 0x7f9cdc098670 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 10 monclient: _renew_subs 2023-09-30T00:46:05.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cdc09f230 con 0x7f9cdc098670 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cd0011cc0 con 0x7f9cdc098670 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cd8ff9700 10 monclient: dump: 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:05.090 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9cdaffd700 4 set_mon_vals no callback set 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 10 monclient: _renew_subs 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.073+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cdc09e460 con 0x7f9cdc098670 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9cd0012630 con 0x7f9cdc098670 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9cd8ff9700 1 --2- 172.21.15.6:0/399917343 >> v2:172.21.15.22:6812/142603 conn(0x7f9cc00357a0 0x7f9cc0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:05.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9cd0047450 con 0x7f9cdc098670 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9ce3153700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9ce3153700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9ce3153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cdc09e460 con 0x7f9cdc098670 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9cdbfff700 1 --2- 172.21.15.6:0/399917343 >> v2:172.21.15.22:6812/142603 conn(0x7f9cc00357a0 0x7f9cc0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.074+0000 7f9cdbfff700 10 monclient: get_auth_request con 0x7f9cc00357a0 auth_method 0 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.077+0000 7f9cdbfff700 1 --2- 172.21.15.6:0/399917343 >> v2:172.21.15.22:6812/142603 conn(0x7f9cc00357a0 0x7f9cc0037c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc09f9a0 tx=0x7f9cc80091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.079+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9cdc09e460 con 0x7f9cdc098670 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.079+0000 7f9cd8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:05.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.079+0000 7f9cd8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9ce3153700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9ce3153700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9ce3153700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:05.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9cdc09e460 con 0x7f9cdc098670 2023-09-30T00:46:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9cd8ff9700 1 -- 172.21.15.6:0/399917343 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9cdc09e460 con 0x7f9cdc098670 2023-09-30T00:46:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9cd8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:05.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.328+0000 7f9cd8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:05.330 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:05.331 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 >> v2:172.21.15.22:6812/142603 conn(0x7f9cc00357a0 msgr2=0x7f9cc0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 --2- 172.21.15.6:0/399917343 >> v2:172.21.15.22:6812/142603 conn(0x7f9cc00357a0 0x7f9cc0037c50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9cdc09f9a0 tx=0x7f9cc80091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 10 monclient: shutdown 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 msgr2=0x7f9cdc12f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 0x7f9cdc12f110 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9cd000c600 tx=0x7f9cd0007f60 comp rx=0 tx=0).stop 2023-09-30T00:46:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 shutdown_connections 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 --2- 172.21.15.6:0/399917343 >> v2:172.21.15.22:6812/142603 conn(0x7f9cc00357a0 0x7f9cc0037c50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9cdc137b60 0x7f9cdc1369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9cdc098670 0x7f9cdc12f110 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 --2- 172.21.15.6:0/399917343 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9cdc132d00 0x7f9cdc09ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 >> 172.21.15.6:0/399917343 conn(0x7f9cdc0a5480 msgr2=0x7f9cdc12fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 shutdown_connections 2023-09-30T00:46:05.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:05.331+0000 7f9ce3153700 1 -- 172.21.15.6:0/399917343 wait complete. 2023-09-30T00:46:08.346 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:08.485 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.486+0000 7f38d34a9700 10 monclient: get_monmap_and_config 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.486+0000 7f38d34a9700 10 monclient: build_initial_monmap 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.486+0000 7f38d34a9700 10 monclient: monmap: 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:08.486396+0000 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:08.486396+0000 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:08.486 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:08.487 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:08.487 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:08.487 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:08.487 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.486+0000 7f38d34a9700 1 Processor -- start 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 1 -- start start 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient: init 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38cc098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 0x7f38cc137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): picked mon.noname-b con 0x7f38cc132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f38cc137b60 0x7f38cc139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:08.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): picked mon.noname-c con 0x7f38cc137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc0a1000 con 0x7f38cc098670 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38cc099150 con 0x7f38cc137b60 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc0a0a50 con 0x7f38cc132d00 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d34a9700 10 monclient(hunting): authenticate will time out at 1364.784679s 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 0x7f38cc137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 0x7f38cc137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:48824/0 (socket says 172.21.15.6:48824) 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 1 -- 172.21.15.6:0/1630027235 learned_addr learned my addr 172.21.15.6:0/1630027235 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): get_auth_request con 0x7f38cc132d00 auth_method 0 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:08.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d1a46700 1 --2- 172.21.15.6:0/1630027235 >> v2:172.21.15.22:3300/0 conn(0x7f38cc137b60 0x7f38cc139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.487+0000 7f38d0a44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): get_auth_request con 0x7f38cc137b60 auth_method 0 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:08.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d1a46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 10 monclient(hunting): handle_auth_done global_id 4460 payload 274 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 1 -- 172.21.15.6:0/1630027235 >> v2:172.21.15.22:3300/0 conn(0x7f38cc137b60 msgr2=0x7f38cc139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 1 --2- 172.21.15.6:0/1630027235 >> v2:172.21.15.22:3300/0 conn(0x7f38cc137b60 0x7f38cc139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 1 -- 172.21.15.6:0/1630027235 >> v1:172.21.15.22:6789/0 conn(0x7f38cc098670 legacy=0x7f38cc098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 10 monclient: _finish_hunting 0 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 1 monclient: found mon.noname-b 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.488+0000 7f38d0a44700 1 -- 172.21.15.6:0/1630027235 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc0a0730 con 0x7f38cc132d00 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38d0a44700 1 -- 172.21.15.6:0/1630027235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 msgr2=0x7f38cc137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:08.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38d0a44700 1 --2- 172.21.15.6:0/1630027235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 0x7f38cc137620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0099b0 tx=0x7f38bc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1266b0eede39c69d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 1 -- 172.21.15.6:0/1630027235 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc009ea0 con 0x7f38cc132d00 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: dump: 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.498 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: _finish_auth 0 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 1 -- 172.21.15.6:0/1630027235 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc00ba30 con 0x7f38cc132d00 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 1 -- 172.21.15.6:0/1630027235 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38bc00bdf0 con 0x7f38cc132d00 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:08.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38c3fff700 10 monclient: dump: 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38d34a9700 5 monclient: authenticate success, global_id 4460 2023-09-30T00:46:08.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38d34a9700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.490+0000 7f38d34a9700 4 set_mon_vals no callback set 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 10 monclient: shutdown 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 -- 172.21.15.6:0/1630027235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 msgr2=0x7f38cc137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 --2- 172.21.15.6:0/1630027235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 0x7f38cc137620 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f38bc0099b0 tx=0x7f38bc009250 comp rx=0 tx=0).stop 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 -- 172.21.15.6:0/1630027235 shutdown_connections 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 --2- 172.21.15.6:0/1630027235 >> v2:172.21.15.22:3300/0 conn(0x7f38cc137b60 0x7f38cc139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 --2- 172.21.15.6:0/1630027235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc132d00 0x7f38cc137620 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 -- 172.21.15.6:0/1630027235 >> 172.21.15.6:0/1630027235 conn(0x7f38cc0a5480 msgr2=0x7f38cc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 -- 172.21.15.6:0/1630027235 shutdown_connections 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.491+0000 7f38d34a9700 1 -- 172.21.15.6:0/1630027235 wait complete. 2023-09-30T00:46:08.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.492+0000 7f38d34a9700 10 monclient: build_initial_monmap 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.492+0000 7f38d34a9700 10 monclient: monmap: 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:08.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.492+0000 7f38d34a9700 1 Processor -- start 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.492+0000 7f38d34a9700 1 -- start start 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.492+0000 7f38d34a9700 10 monclient: init 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38cc098670 0x7f38cc09b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): picked mon.noname-b con 0x7f38cc098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38cc132d00 0x7f38cc09c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): picked mon.noname-a con 0x7f38cc132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 0x7f38cc09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): picked mon.noname-c con 0x7f38cc137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc0a1500 con 0x7f38cc132d00 2023-09-30T00:46:08.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc137810 con 0x7f38cc098670 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc136350 con 0x7f38cc137b60 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d34a9700 10 monclient(hunting): authenticate will time out at 1364.790204s 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 0x7f38cc09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38cc132d00 0x7f38cc09c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 0x7f38cc09cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:48828/0 (socket says 172.21.15.6:48828) 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38cc132d00 0x7f38cc09c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58358/0 (socket says 172.21.15.6:58358) 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 -- 172.21.15.6:0/3864647850 learned_addr learned my addr 172.21.15.6:0/3864647850 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:08.504 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): get_auth_request con 0x7f38cc137b60 auth_method 0 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): get_auth_request con 0x7f38cc132d00 auth_method 0 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38cc098670 0x7f38cc09b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): get_auth_request con 0x7f38cc098670 auth_method 0 2023-09-30T00:46:08.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d0a44700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1245700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient(hunting): handle_auth_done global_id 4463 payload 824 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 -- 172.21.15.6:0/3864647850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38cc098670 msgr2=0x7f38cc09b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38cc098670 0x7f38cc09b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 -- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38cc132d00 msgr2=0x7f38cc09c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38cc132d00 0x7f38cc09c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient: _finish_hunting 0 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 monclient: found mon.noname-c 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.493+0000 7f38d1a46700 1 -- 172.21.15.6:0/3864647850 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38bc0094f0 con 0x7f38cc137b60 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.495+0000 7f38d1a46700 1 -- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 msgr2=0x7f38cc09cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.495+0000 7f38d1a46700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 0x7f38cc09cce0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009ec0 tx=0x7f38c400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c400cd80 con 0x7f38cc137b60 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: dump: 2023-09-30T00:46:08.507 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: _finish_auth 0 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:08.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c400f040 con 0x7f38cc137b60 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 5 monclient: authenticate success, global_id 4463 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c40146d0 con 0x7f38cc137b60 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 10 monclient: _renew_subs 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc133b10 con 0x7f38cc137b60 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c37fe700 4 set_mon_vals no callback set 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38c17fa700 10 monclient: dump: 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:08.509 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 10 monclient: _renew_subs 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.496+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc09e5c0 con 0x7f38cc137b60 2023-09-30T00:46:08.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f38c4014a80 con 0x7f38cc137b60 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38d34a9700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38d34a9700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38d34a9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc09e840 con 0x7f38cc137b60 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38c17fa700 1 --2- 172.21.15.6:0/3864647850 >> v2:172.21.15.22:6812/142603 conn(0x7f38b00357a0 0x7f38b0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f38c4048660 con 0x7f38cc137b60 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38d1245700 1 --2- 172.21.15.6:0/3864647850 >> v2:172.21.15.22:6812/142603 conn(0x7f38b00357a0 0x7f38b0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:08.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.497+0000 7f38d1245700 10 monclient: get_auth_request con 0x7f38b00357a0 auth_method 0 2023-09-30T00:46:08.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.500+0000 7f38d1245700 1 --2- 172.21.15.6:0/3864647850 >> v2:172.21.15.22:6812/142603 conn(0x7f38b00357a0 0x7f38b0037c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f38c80039a0 tx=0x7f38c8007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:08.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.502+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38c4012b70 con 0x7f38cc137b60 2023-09-30T00:46:08.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.502+0000 7f38c17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:08.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.502+0000 7f38c17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:08.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.751+0000 7f38d34a9700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.751+0000 7f38d34a9700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.751+0000 7f38d34a9700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.751+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f38cc072b00 con 0x7f38cc137b60 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.752+0000 7f38c17fa700 1 -- 172.21.15.6:0/3864647850 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f38c4018080 con 0x7f38cc137b60 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.752+0000 7f38c17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.752+0000 7f38c17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:08.752 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:08.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 >> v2:172.21.15.22:6812/142603 conn(0x7f38b00357a0 msgr2=0x7f38b0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 --2- 172.21.15.6:0/3864647850 >> v2:172.21.15.22:6812/142603 conn(0x7f38b00357a0 0x7f38b0037c50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f38c80039a0 tx=0x7f38c8007ea0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 10 monclient: shutdown 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 msgr2=0x7f38cc09cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 0x7f38cc09cce0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f38c4009ec0 tx=0x7f38c400b040 comp rx=0 tx=0).stop 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 shutdown_connections 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 --2- 172.21.15.6:0/3864647850 >> v2:172.21.15.22:6812/142603 conn(0x7f38b00357a0 0x7f38b0037c50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38cc137b60 0x7f38cc09cce0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38cc098670 0x7f38cc09b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 --2- 172.21.15.6:0/3864647850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38cc132d00 0x7f38cc09c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 >> 172.21.15.6:0/3864647850 conn(0x7f38cc0a5480 msgr2=0x7f38cc131b10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:08.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 shutdown_connections 2023-09-30T00:46:08.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:08.754+0000 7f38d34a9700 1 -- 172.21.15.6:0/3864647850 wait complete. 2023-09-30T00:46:11.768 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.905+0000 7fef9e9b1700 10 monclient: get_monmap_and_config 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.905+0000 7fef9e9b1700 10 monclient: build_initial_monmap 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.905+0000 7fef9e9b1700 10 monclient: monmap: 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:11.905464+0000 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:11.905464+0000 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:11.905 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:11.906 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:11.906 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:11.906 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:11.906 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:11.906 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:11.906 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 1 Processor -- start 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 1 -- start start 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient: init 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient: _add_conns ranks=[0,1,3,2,4] 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fef98098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:11.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fef98132d00 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fef98138d40 0x7fef9813b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient(hunting): picked mon.noname-a con 0x7fef98138d40 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef980a1000 con 0x7fef98098670 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef980a0730 con 0x7fef98132d00 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef98099150 con 0x7fef98138d40 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9e9b1700 10 monclient(hunting): authenticate will time out at 1368.203614s 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef97fff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fef98098670 0x7fef98098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42820/0 (socket says 172.21.15.6:42820) 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef97fff700 1 -- 172.21.15.6:0/1913968818 learned_addr learned my addr 172.21.15.6:0/1913968818 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 1 --2- 172.21.15.6:0/1913968818 >> v2:172.21.15.6:3300/0 conn(0x7fef98138d40 0x7fef9813b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:11.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): get_auth_request con 0x7fef98138d40 auth_method 0 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.906+0000 7fef9cf4e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 10 monclient(hunting): handle_auth_done global_id 4458 payload 274 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 1 -- 172.21.15.6:0/1913968818 >> v1:172.21.15.22:6789/0 conn(0x7fef98132d00 legacy=0x7fef98137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 1 -- 172.21.15.6:0/1913968818 >> v1:172.21.15.6:6789/0 conn(0x7fef98098670 legacy=0x7fef98098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 10 monclient: _finish_hunting 0 2023-09-30T00:46:11.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 1 monclient: found mon.noname-a 2023-09-30T00:46:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:11.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.907+0000 7fef9cf4e700 1 -- 172.21.15.6:0/1913968818 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef98068df0 con 0x7fef98138d40 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef9cf4e700 1 -- 172.21.15.6:0/1913968818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98138d40 msgr2=0x7fef9813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef9cf4e700 1 --2- 172.21.15.6:0/1913968818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98138d40 0x7fef9813b120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef90009e60 tx=0x7fef90009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd6ee7fc8304576 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 1 -- 172.21.15.6:0/1913968818 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef9000e070 con 0x7fef98138d40 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: dump: 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:11.917 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 1 -- 172.21.15.6:0/1913968818 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef90012430 con 0x7fef98138d40 2023-09-30T00:46:11.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef9e9b1700 5 monclient: authenticate success, global_id 4458 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef9e9b1700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef9e9b1700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 1 -- 172.21.15.6:0/1913968818 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef900127d0 con 0x7fef98138d40 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef9e9b1700 4 set_mon_vals no callback set 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.909+0000 7fef96ffd700 10 monclient: dump: 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.919 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 10 monclient: shutdown 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1913968818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98138d40 msgr2=0x7fef9813b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1913968818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98138d40 0x7fef9813b120 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fef90009e60 tx=0x7fef90009780 comp rx=0 tx=0).stop 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1913968818 shutdown_connections 2023-09-30T00:46:11.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1913968818 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98138d40 0x7fef9813b120 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1913968818 >> 172.21.15.6:0/1913968818 conn(0x7fef980a5480 msgr2=0x7fef980a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1913968818 shutdown_connections 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1913968818 wait complete. 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 10 monclient: build_initial_monmap 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.910+0000 7fef9e9b1700 10 monclient: monmap: 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:11.921 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 Processor -- start 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 -- start start 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient: init 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 0x7fef981415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): picked mon.noname-a con 0x7fef98098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:11.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fef98132d00 0x7fef98138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): picked mon.noname-c con 0x7fef98132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef98138d40 0x7fef9813f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): picked mon.noname-b con 0x7fef98138d40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef9812f420 con 0x7fef98098670 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef98130320 con 0x7fef98138d40 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef9812e0e0 con 0x7fef98132d00 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:11.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef9e9b1700 10 monclient(hunting): authenticate will time out at 1368.208901s 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.911+0000 7fef97fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 0x7fef981415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fef98132d00 0x7fef98138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 0x7fef981415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58370/0 (socket says 172.21.15.6:58370) 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 -- 172.21.15.6:0/1804065998 learned_addr learned my addr 172.21.15.6:0/1804065998 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): get_auth_request con 0x7fef98098670 auth_method 0 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): get_auth_request con 0x7fef98132d00 auth_method 0 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:11.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef9cf4e700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef98138d40 0x7fef9813f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef9cf4e700 10 monclient(hunting): get_auth_request con 0x7fef98138d40 auth_method 0 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef9cf4e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef9cf4e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef9cf4e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:11.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient(hunting): handle_auth_done global_id 4461 payload 824 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 -- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fef98132d00 msgr2=0x7fef98138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fef98132d00 0x7fef98138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 -- 172.21.15.6:0/1804065998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef98138d40 msgr2=0x7fef9813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef98138d40 0x7fef9813f4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 monclient: found mon.noname-a 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef97fff700 1 -- 172.21.15.6:0/1804065998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef90009a20 con 0x7fef98098670 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.912+0000 7fef977fe700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fef98132d00 0x7fef98138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:11.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef97fff700 1 -- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 msgr2=0x7fef981415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef97fff700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 0x7fef981415c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c5e0 tx=0x7fef88004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88005d40 con 0x7fef98098670 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: dump: 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:11.927 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: _finish_auth 0 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 5 monclient: authenticate success, global_id 4461 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef880070d0 con 0x7fef98098670 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 10 monclient: _renew_subs 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef981335b0 con 0x7fef98098670 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fef88007490 con 0x7fef98098670 2023-09-30T00:46:11.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef7ffff700 10 monclient: dump: 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:11.929 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 10 monclient: _renew_subs 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef967fc700 4 set_mon_vals no callback set 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.914+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef9812e760 con 0x7fef98098670 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.915+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fef8801b3b0 con 0x7fef98098670 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.915+0000 7fef9e9b1700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.915+0000 7fef9e9b1700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.915+0000 7fef9e9b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.915+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef98072b00 con 0x7fef98098670 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.915+0000 7fef7ffff700 1 --2- 172.21.15.6:0/1804065998 >> v2:172.21.15.22:6812/142603 conn(0x7fef840357a0 0x7fef84037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.916+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fef88046f60 con 0x7fef98098670 2023-09-30T00:46:11.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.916+0000 7fef977fe700 1 --2- 172.21.15.6:0/1804065998 >> v2:172.21.15.22:6812/142603 conn(0x7fef840357a0 0x7fef84037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.916+0000 7fef977fe700 10 monclient: get_auth_request con 0x7fef840357a0 auth_method 0 2023-09-30T00:46:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.918+0000 7fef977fe700 1 --2- 172.21.15.6:0/1804065998 >> v2:172.21.15.22:6812/142603 conn(0x7fef840357a0 0x7fef84037c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fef98142650 tx=0x7fef8c007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.920+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fef8801be60 con 0x7fef98098670 2023-09-30T00:46:11.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.921+0000 7fef7ffff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:11.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:11.921+0000 7fef7ffff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:12.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.166+0000 7fef9e9b1700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.166+0000 7fef9e9b1700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.166+0000 7fef9e9b1700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.166+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fef98133e20 con 0x7fef98098670 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.167+0000 7fef7ffff700 1 -- 172.21.15.6:0/1804065998 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fef88017070 con 0x7fef98098670 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.167+0000 7fef7ffff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.167+0000 7fef7ffff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:12.167 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:12.168 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 >> v2:172.21.15.22:6812/142603 conn(0x7fef840357a0 msgr2=0x7fef84037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1804065998 >> v2:172.21.15.22:6812/142603 conn(0x7fef840357a0 0x7fef84037c50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fef98142650 tx=0x7fef8c007ea0 comp rx=0 tx=0).stop 2023-09-30T00:46:12.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 10 monclient: shutdown 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 msgr2=0x7fef981415c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 0x7fef981415c0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fef8800c5e0 tx=0x7fef88004a80 comp rx=0 tx=0).stop 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 shutdown_connections 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1804065998 >> v2:172.21.15.22:6812/142603 conn(0x7fef840357a0 0x7fef84037c50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fef98138d40 0x7fef9813f4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fef98098670 0x7fef981415c0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 --2- 172.21.15.6:0/1804065998 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fef98132d00 0x7fef98138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 >> 172.21.15.6:0/1804065998 conn(0x7fef980a5480 msgr2=0x7fef980a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 shutdown_connections 2023-09-30T00:46:12.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:12.169+0000 7fef9e9b1700 1 -- 172.21.15.6:0/1804065998 wait complete. 2023-09-30T00:46:15.183 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:15.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.323+0000 7fa30267a700 10 monclient: get_monmap_and_config 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.323+0000 7fa30267a700 10 monclient: build_initial_monmap 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.323+0000 7fa30267a700 10 monclient: monmap: 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:15.323281+0000 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:15.323281+0000 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:15.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:15.324 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:15.324 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.323+0000 7fa30267a700 1 Processor -- start 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.323+0000 7fa30267a700 1 -- start start 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.323+0000 7fa30267a700 10 monclient: init 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa2fc098670 0x7fa2fc132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:15.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): picked mon.noname-c con 0x7fa2fc098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): picked mon.noname-b con 0x7fa2fc098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa2fc098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2fc0a1000 con 0x7fa2fc098e90 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2fc0a0a50 con 0x7fa2fc098670 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc0a0730 con 0x7fa2fc098a40 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa30267a700 10 monclient(hunting): authenticate will time out at 1371.621432s 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:15.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:48838/0 (socket says 172.21.15.6:48838) 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 1 -- 172.21.15.6:0/3820094837 learned_addr learned my addr 172.21.15.6:0/3820094837 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): get_auth_request con 0x7fa2fc098a40 auth_method 0 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 1 --2- 172.21.15.6:0/3820094837 >> v2:172.21.15.22:3300/0 conn(0x7fa2fc098670 0x7fa2fc132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): get_auth_request con 0x7fa2fc098670 auth_method 0 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:15.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_done global_id 4469 payload 274 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 1 -- 172.21.15.6:0/3820094837 >> v2:172.21.15.22:3300/0 conn(0x7fa2fc098670 msgr2=0x7fa2fc132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 1 --2- 172.21.15.6:0/3820094837 >> v2:172.21.15.22:3300/0 conn(0x7fa2fc098670 0x7fa2fc132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.324+0000 7fa2fb7fe700 1 -- 172.21.15.6:0/3820094837 >> v1:172.21.15.22:6789/0 conn(0x7fa2fc098e90 legacy=0x7fa2fc12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.325+0000 7fa2fb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.325+0000 7fa2fb7fe700 1 monclient: found mon.noname-b 2023-09-30T00:46:15.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.325+0000 7fa2fb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:15.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.325+0000 7fa2fb7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:15.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.325+0000 7fa2fb7fe700 1 -- 172.21.15.6:0/3820094837 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2fc068df0 con 0x7fa2fc098a40 2023-09-30T00:46:15.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2fb7fe700 1 -- 172.21.15.6:0/3820094837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 msgr2=0x7fa2fc12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:15.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2fb7fe700 1 --2- 172.21.15.6:0/3820094837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12ede0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00099b0 tx=0x7fa2f0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=807d44923e4ed78b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:15.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 1 -- 172.21.15.6:0/3820094837 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f0009ea0 con 0x7fa2fc098a40 2023-09-30T00:46:15.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:15.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:15.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: dump: 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:15.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 5 monclient: authenticate success, global_id 4469 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 1 -- 172.21.15.6:0/3820094837 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f000ba30 con 0x7fa2fc098a40 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 4 set_mon_vals no callback set 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 1 -- 172.21.15.6:0/3820094837 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2f000bdf0 con 0x7fa2fc098a40 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa2faffd700 10 monclient: dump: 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:15.338 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:15.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 10 monclient: shutdown 2023-09-30T00:46:15.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:15.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 1 -- 172.21.15.6:0/3820094837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 msgr2=0x7fa2fc12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:15.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.327+0000 7fa30267a700 1 --2- 172.21.15.6:0/3820094837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12ede0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa2f00099b0 tx=0x7fa2f0009250 comp rx=0 tx=0).stop 2023-09-30T00:46:15.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 -- 172.21.15.6:0/3820094837 shutdown_connections 2023-09-30T00:46:15.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 --2- 172.21.15.6:0/3820094837 >> v2:172.21.15.22:3300/0 conn(0x7fa2fc098670 0x7fa2fc132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 --2- 172.21.15.6:0/3820094837 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12ede0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 -- 172.21.15.6:0/3820094837 >> 172.21.15.6:0/3820094837 conn(0x7fa2fc0a5480 msgr2=0x7fa2fc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:15.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 -- 172.21.15.6:0/3820094837 shutdown_connections 2023-09-30T00:46:15.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 -- 172.21.15.6:0/3820094837 wait complete. 2023-09-30T00:46:15.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 10 monclient: build_initial_monmap 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 10 monclient: monmap: 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:15.342 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.328+0000 7fa30267a700 1 Processor -- start 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 -- start start 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient: init 2023-09-30T00:46:15.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 0x7fa2fc141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): picked mon.noname-a con 0x7fa2fc098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): picked mon.noname-c con 0x7fa2fc098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2fc098e90 0x7fa2fc09c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): picked mon.noname-b con 0x7fa2fc098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:15.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc135390 con 0x7fa2fc098670 2023-09-30T00:46:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc12da70 con 0x7fa2fc098e90 2023-09-30T00:46:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:15.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc1429e0 con 0x7fa2fc098a40 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa30267a700 10 monclient(hunting): authenticate will time out at 1371.626705s 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 0x7fa2fc141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:48840/0 (socket says 172.21.15.6:48840) 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 0x7fa2fc141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58374/0 (socket says 172.21.15.6:58374) 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 1 -- 172.21.15.6:0/3898347253 learned_addr learned my addr 172.21.15.6:0/3898347253 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 10 monclient(hunting): get_auth_request con 0x7fa2fc098a40 auth_method 0 2023-09-30T00:46:15.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): get_auth_request con 0x7fa2fc098670 auth_method 0 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa300c17700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2fc098e90 0x7fa2fc09c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa300c17700 10 monclient(hunting): get_auth_request con 0x7fa2fc098e90 auth_method 0 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa300c17700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa300c17700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa300c17700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:15.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.329+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa300c17700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa300c17700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa300c17700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 10 monclient(hunting): handle_auth_done global_id 4464 payload 824 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 1 -- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 msgr2=0x7fa2fc12f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 1 -- 172.21.15.6:0/3898347253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2fc098e90 msgr2=0x7fa2fc09c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2fc098e90 0x7fa2fc09c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 1 monclient: found mon.noname-a 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.330+0000 7fa2fbfff700 1 -- 172.21.15.6:0/3898347253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f00094f0 con 0x7fa2fc098670 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2fbfff700 1 -- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 msgr2=0x7fa2fc141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2fbfff700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 0x7fa2fc141b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00ea50 tx=0x7fa2ec0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec0095d0 con 0x7fa2fc098670 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: dump: 2023-09-30T00:46:15.349 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: _finish_auth 0 2023-09-30T00:46:15.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2ec00ca00 con 0x7fa2fc098670 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 5 monclient: authenticate success, global_id 4464 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 10 monclient: _renew_subs 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc09f010 con 0x7fa2fc098670 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ec00cdc0 con 0x7fa2fc098670 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2e3fff700 10 monclient: dump: 2023-09-30T00:46:15.351 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa2fa7fc700 4 set_mon_vals no callback set 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 10 monclient: _renew_subs 2023-09-30T00:46:15.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.332+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc09e370 con 0x7fa2fc098670 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa2ec013680 con 0x7fa2fc098670 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa30267a700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa30267a700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa30267a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc072b00 con 0x7fa2fc098670 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa2e3fff700 1 --2- 172.21.15.6:0/3898347253 >> v2:172.21.15.22:6812/142603 conn(0x7fa2e80357a0 0x7fa2e8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.333+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa2ec047bc0 con 0x7fa2fc098670 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.334+0000 7fa2fb7fe700 1 --2- 172.21.15.6:0/3898347253 >> v2:172.21.15.22:6812/142603 conn(0x7fa2e80357a0 0x7fa2e8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.334+0000 7fa2fb7fe700 10 monclient: get_auth_request con 0x7fa2e80357a0 auth_method 0 2023-09-30T00:46:15.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.336+0000 7fa2fb7fe700 1 --2- 172.21.15.6:0/3898347253 >> v2:172.21.15.22:6812/142603 conn(0x7fa2e80357a0 0x7fa2e8037c50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc09e060 tx=0x7fa2f0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.338+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa2ec0123d0 con 0x7fa2fc098670 2023-09-30T00:46:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.338+0000 7fa2e3fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:15.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.338+0000 7fa2e3fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:15.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.588+0000 7fa30267a700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.588+0000 7fa30267a700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.588+0000 7fa30267a700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.588+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa2fc135150 con 0x7fa2fc098670 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.589+0000 7fa2e3fff700 1 -- 172.21.15.6:0/3898347253 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa2ec01b3b0 con 0x7fa2fc098670 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.589+0000 7fa2e3fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.589+0000 7fa2e3fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:15.589 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:15.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 >> v2:172.21.15.22:6812/142603 conn(0x7fa2e80357a0 msgr2=0x7fa2e8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:15.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 --2- 172.21.15.6:0/3898347253 >> v2:172.21.15.22:6812/142603 conn(0x7fa2e80357a0 0x7fa2e8037c50 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fa2fc09e060 tx=0x7fa2f0016000 comp rx=0 tx=0).stop 2023-09-30T00:46:15.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 10 monclient: shutdown 2023-09-30T00:46:15.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:15.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 msgr2=0x7fa2fc141b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 0x7fa2fc141b70 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa2ec00ea50 tx=0x7fa2ec0091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 shutdown_connections 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 --2- 172.21.15.6:0/3898347253 >> v2:172.21.15.22:6812/142603 conn(0x7fa2e80357a0 0x7fa2e8037c50 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa2fc098e90 0x7fa2fc09c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa2fc098670 0x7fa2fc141b70 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 --2- 172.21.15.6:0/3898347253 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa2fc098a40 0x7fa2fc12f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 >> 172.21.15.6:0/3898347253 conn(0x7fa2fc0a5480 msgr2=0x7fa2fc069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 shutdown_connections 2023-09-30T00:46:15.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:15.591+0000 7fa30267a700 1 -- 172.21.15.6:0/3898347253 wait complete. 2023-09-30T00:46:18.604 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.744+0000 7fc290b31700 10 monclient: get_monmap_and_config 2023-09-30T00:46:18.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.744+0000 7fc290b31700 10 monclient: build_initial_monmap 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.744+0000 7fc290b31700 10 monclient: monmap: 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:18.745143+0000 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:18.745143+0000 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:18.745 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:18.746 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.745+0000 7fc290b31700 1 Processor -- start 2023-09-30T00:46:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.745+0000 7fc290b31700 1 -- start start 2023-09-30T00:46:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.745+0000 7fc290b31700 10 monclient: init 2023-09-30T00:46:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T00:46:18.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc28c098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc28c132d00 0x7fc28c137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): picked mon.noname-a con 0x7fc28c132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc28c137b60 0x7fc28c139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): picked mon.noname-c con 0x7fc28c137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc28c0a1000 con 0x7fc28c098670 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c099150 con 0x7fc28c132d00 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc28c0a0a50 con 0x7fc28c137b60 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:18.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc290b31700 10 monclient(hunting): authenticate will time out at 1375.043457s 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc28c132d00 0x7fc28c137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc28c132d00 0x7fc28c137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33896/0 (socket says 172.21.15.6:33896) 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 -- 172.21.15.6:0/998010556 learned_addr learned my addr 172.21.15.6:0/998010556 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): get_auth_request con 0x7fc28c132d00 auth_method 0 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 1 --2- 172.21.15.6:0/998010556 >> v2:172.21.15.22:3300/0 conn(0x7fc28c137b60 0x7fc28c139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): get_auth_request con 0x7fc28c137b60 auth_method 0 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:18.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc28ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient(hunting): handle_auth_done global_id 4467 payload 274 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 -- 172.21.15.6:0/998010556 >> v2:172.21.15.22:3300/0 conn(0x7fc28c137b60 msgr2=0x7fc28c139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 --2- 172.21.15.6:0/998010556 >> v2:172.21.15.22:3300/0 conn(0x7fc28c137b60 0x7fc28c139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 -- 172.21.15.6:0/998010556 >> v1:172.21.15.22:6789/0 conn(0x7fc28c098670 legacy=0x7fc28c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:46:18.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 monclient: found mon.noname-a 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.746+0000 7fc289d9b700 1 -- 172.21.15.6:0/998010556 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c0a0730 con 0x7fc28c132d00 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc289d9b700 1 -- 172.21.15.6:0/998010556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 msgr2=0x7fc28c137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc289d9b700 1 --2- 172.21.15.6:0/998010556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c137620 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0a3330 tx=0x7fc274009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6182c272994e78dd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 1 -- 172.21.15.6:0/998010556 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc274009e70 con 0x7fc28c132d00 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: dump: 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:18.757 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: _finish_auth 0 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 1 -- 172.21.15.6:0/998010556 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27400ba30 con 0x7fc28c132d00 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 5 monclient: authenticate success, global_id 4467 2023-09-30T00:46:18.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 1 -- 172.21.15.6:0/998010556 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27400bdf0 con 0x7fc28c132d00 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 4 set_mon_vals no callback set 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc28959a700 10 monclient: dump: 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:18.759 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 10 monclient: shutdown 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:18.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 1 -- 172.21.15.6:0/998010556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 msgr2=0x7fc28c137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.749+0000 7fc290b31700 1 --2- 172.21.15.6:0/998010556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c137620 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0a3330 tx=0x7fc274009250 comp rx=0 tx=0).stop 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 -- 172.21.15.6:0/998010556 shutdown_connections 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 --2- 172.21.15.6:0/998010556 >> v2:172.21.15.22:3300/0 conn(0x7fc28c137b60 0x7fc28c139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 --2- 172.21.15.6:0/998010556 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c137620 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 -- 172.21.15.6:0/998010556 >> 172.21.15.6:0/998010556 conn(0x7fc28c0a5480 msgr2=0x7fc28c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 -- 172.21.15.6:0/998010556 shutdown_connections 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 -- 172.21.15.6:0/998010556 wait complete. 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 10 monclient: build_initial_monmap 2023-09-30T00:46:18.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 10 monclient: monmap: 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:18.762 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.750+0000 7fc290b31700 1 Processor -- start 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 -- start start 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient: init 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 0x7fc28c099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): picked mon.noname-c con 0x7fc28c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c0997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): picked mon.noname-a con 0x7fc28c132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc28c137b60 0x7fc28c099d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): picked mon.noname-b con 0x7fc28c137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c136630 con 0x7fc28c132d00 2023-09-30T00:46:18.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c0a1500 con 0x7fc28c137b60 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc28c1378a0 con 0x7fc28c098670 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc290b31700 10 monclient(hunting): authenticate will time out at 1375.048502s 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 0x7fc28c099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c0997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 0x7fc28c099290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42932/0 (socket says 172.21.15.6:42932) 2023-09-30T00:46:18.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c0997d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33900/0 (socket says 172.21.15.6:33900) 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 1 -- 172.21.15.6:0/304834761 learned_addr learned my addr 172.21.15.6:0/304834761 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): get_auth_request con 0x7fc28c098670 auth_method 0 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): get_auth_request con 0x7fc28c132d00 auth_method 0 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc28c137b60 0x7fc28c099d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): get_auth_request con 0x7fc28c137b60 auth_method 0 2023-09-30T00:46:18.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc289d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:18.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 10 monclient(hunting): handle_auth_done global_id 4475 payload 824 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 1 -- 172.21.15.6:0/304834761 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc28c137b60 msgr2=0x7fc28c099d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.751+0000 7fc28a59c700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc28c137b60 0x7fc28c099d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 1 -- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 msgr2=0x7fc28c0997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c0997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 1 monclient: found mon.noname-c 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:18.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.752+0000 7fc28a59c700 1 -- 172.21.15.6:0/304834761 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2740094f0 con 0x7fc28c098670 2023-09-30T00:46:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc28a59c700 1 -- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 msgr2=0x7fc28c099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc28a59c700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 0x7fc28c099290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00c600 tx=0x7fc27c007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:18.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c0117a0 con 0x7fc28c098670 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: dump: 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:18.770 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc27c011900 con 0x7fc28c098670 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc27c011cc0 con 0x7fc28c098670 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc2827fc700 10 monclient: dump: 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:18.771 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 5 monclient: authenticate success, global_id 4475 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 10 monclient: _renew_subs 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:18.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28c133680 con 0x7fc28c098670 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc288d99700 4 set_mon_vals no callback set 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 10 monclient: _renew_subs 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.754+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28c09a670 con 0x7fc28c098670 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc27c012630 con 0x7fc28c098670 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc290b31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc290b31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc290b31700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28c09ac30 con 0x7fc28c098670 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc2827fc700 1 --2- 172.21.15.6:0/304834761 >> v2:172.21.15.22:6812/142603 conn(0x7fc26c0357a0 0x7fc26c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.755+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc27c047650 con 0x7fc28c098670 2023-09-30T00:46:18.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.756+0000 7fc289d9b700 1 --2- 172.21.15.6:0/304834761 >> v2:172.21.15.22:6812/142603 conn(0x7fc26c0357a0 0x7fc26c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.756+0000 7fc289d9b700 10 monclient: get_auth_request con 0x7fc26c0357a0 auth_method 0 2023-09-30T00:46:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.758+0000 7fc289d9b700 1 --2- 172.21.15.6:0/304834761 >> v2:172.21.15.22:6812/142603 conn(0x7fc26c0357a0 0x7fc26c037c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0a3330 tx=0x7fc274016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.760+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc28c09ac30 con 0x7fc28c098670 2023-09-30T00:46:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.760+0000 7fc2827fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:18.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:18.760+0000 7fc2827fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc290b31700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc290b31700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc290b31700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc28c09ac30 con 0x7fc28c098670 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc2827fc700 1 -- 172.21.15.6:0/304834761 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc28c09ac30 con 0x7fc28c098670 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc2827fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:19.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.009+0000 7fc2827fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:19.010 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:19.010 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:19.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 >> v2:172.21.15.22:6812/142603 conn(0x7fc26c0357a0 msgr2=0x7fc26c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 --2- 172.21.15.6:0/304834761 >> v2:172.21.15.22:6812/142603 conn(0x7fc26c0357a0 0x7fc26c037c50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc28c0a3330 tx=0x7fc274016000 comp rx=0 tx=0).stop 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 10 monclient: shutdown 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 msgr2=0x7fc28c099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 0x7fc28c099290 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc27c00c600 tx=0x7fc27c007f60 comp rx=0 tx=0).stop 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 shutdown_connections 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 --2- 172.21.15.6:0/304834761 >> v2:172.21.15.22:6812/142603 conn(0x7fc26c0357a0 0x7fc26c037c50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc28c137b60 0x7fc28c099d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc28c098670 0x7fc28c099290 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 --2- 172.21.15.6:0/304834761 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc28c132d00 0x7fc28c0997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.011+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 >> 172.21.15.6:0/304834761 conn(0x7fc28c0a5480 msgr2=0x7fc28c0a3430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:19.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.012+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 shutdown_connections 2023-09-30T00:46:19.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:19.012+0000 7fc290b31700 1 -- 172.21.15.6:0/304834761 wait complete. 2023-09-30T00:46:22.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:22.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.163+0000 7fc6d9dd6700 10 monclient: get_monmap_and_config 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.164+0000 7fc6d9dd6700 10 monclient: build_initial_monmap 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.164+0000 7fc6d9dd6700 10 monclient: monmap: 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:22.164221+0000 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:22.164221+0000 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:22.164 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:22.165 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:22.165 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:22.165 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:22.165 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.164+0000 7fc6d9dd6700 1 Processor -- start 2023-09-30T00:46:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 1 -- start start 2023-09-30T00:46:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient: init 2023-09-30T00:46:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient: _add_conns ranks=[1,0,3,2,4] 2023-09-30T00:46:22.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc6d4098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc6d4132d00 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6d4138d40 0x7fc6d413b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient(hunting): picked mon.noname-a con 0x7fc6d4138d40 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d40a1000 con 0x7fc6d4132d00 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6d40a0730 con 0x7fc6d4098670 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6d4099150 con 0x7fc6d4138d40 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d9dd6700 10 monclient(hunting): authenticate will time out at 1378.462584s 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6d4138d40 0x7fc6d413b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6cbfff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc6d4132d00 0x7fc6d4137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57706/0 (socket says 172.21.15.6:57706) 2023-09-30T00:46:22.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc6d4138d40 0x7fc6d413b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33916/0 (socket says 172.21.15.6:33916) 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6cbfff700 1 -- 172.21.15.6:0/3370374586 learned_addr learned my addr 172.21.15.6:0/3370374586 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): get_auth_request con 0x7fc6d4138d40 auth_method 0 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.165+0000 7fc6d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 10 monclient(hunting): handle_auth_done global_id 4473 payload 274 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 1 -- 172.21.15.6:0/3370374586 >> v1:172.21.15.22:6789/0 conn(0x7fc6d4098670 legacy=0x7fc6d4098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 1 -- 172.21.15.6:0/3370374586 >> v1:172.21.15.6:6789/0 conn(0x7fc6d4132d00 legacy=0x7fc6d4137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:22.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 1 monclient: found mon.noname-a 2023-09-30T00:46:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:22.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.166+0000 7fc6d3fff700 1 -- 172.21.15.6:0/3370374586 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6d4068df0 con 0x7fc6d4138d40 2023-09-30T00:46:22.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d3fff700 1 -- 172.21.15.6:0/3370374586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4138d40 msgr2=0x7fc6d413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:22.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d3fff700 1 --2- 172.21.15.6:0/3370374586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4138d40 0x7fc6d413b120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009e60 tx=0x7fc6cc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9fd6e7912ca25e0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:22.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 1 -- 172.21.15.6:0/3370374586 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc00e070 con 0x7fc6d4138d40 2023-09-30T00:46:22.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:22.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: dump: 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:22.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 1 -- 172.21.15.6:0/3370374586 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6cc012430 con 0x7fc6d4138d40 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d9dd6700 5 monclient: authenticate success, global_id 4473 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d9dd6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d9dd6700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 1 -- 172.21.15.6:0/3370374586 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6cc0127d0 con 0x7fc6d4138d40 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d9dd6700 4 set_mon_vals no callback set 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.168+0000 7fc6d2ffd700 10 monclient: dump: 2023-09-30T00:46:22.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 10 monclient: shutdown 2023-09-30T00:46:22.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/3370374586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4138d40 msgr2=0x7fc6d413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/3370374586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4138d40 0x7fc6d413b120 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc6cc009e60 tx=0x7fc6cc009780 comp rx=0 tx=0).stop 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/3370374586 shutdown_connections 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/3370374586 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4138d40 0x7fc6d413b120 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/3370374586 >> 172.21.15.6:0/3370374586 conn(0x7fc6d40a5480 msgr2=0x7fc6d40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/3370374586 shutdown_connections 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/3370374586 wait complete. 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 10 monclient: build_initial_monmap 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.169+0000 7fc6d9dd6700 10 monclient: monmap: 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:22.179 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.170+0000 7fc6d9dd6700 1 Processor -- start 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.170+0000 7fc6d9dd6700 1 -- start start 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.170+0000 7fc6d9dd6700 10 monclient: init 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:22.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc6d4098670 0x7fc6d4141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): picked mon.noname-c con 0x7fc6d4098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 0x7fc6d4138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): picked mon.noname-a con 0x7fc6d4132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d4138d40 0x7fc6d413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): picked mon.noname-b con 0x7fc6d4138d40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4130400 con 0x7fc6d4132d00 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d4133ad0 con 0x7fc6d4138d40 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:22.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6d412f5a0 con 0x7fc6d4098670 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d9dd6700 10 monclient(hunting): authenticate will time out at 1378.468363s 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc6d4098670 0x7fc6d4141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 0x7fc6d4138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc6d4098670 0x7fc6d4141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:42942/0 (socket says 172.21.15.6:42942) 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 1 -- 172.21.15.6:0/4220375850 learned_addr learned my addr 172.21.15.6:0/4220375850 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): get_auth_request con 0x7fc6d4132d00 auth_method 0 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:22.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): get_auth_request con 0x7fc6d4098670 auth_method 0 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d3fff700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d4138d40 0x7fc6d413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:22.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d3fff700 10 monclient(hunting): get_auth_request con 0x7fc6d4138d40 auth_method 0 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient(hunting): handle_auth_done global_id 4476 payload 824 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 -- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc6d4098670 msgr2=0x7fc6d4141600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc6d4098670 0x7fc6d4141600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 -- 172.21.15.6:0/4220375850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d4138d40 msgr2=0x7fc6d413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d4138d40 0x7fc6d413f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 monclient: found mon.noname-a 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:22.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6cbfff700 1 -- 172.21.15.6:0/4220375850 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6cc009a20 con 0x7fc6d4132d00 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.171+0000 7fc6d3fff700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d4138d40 0x7fc6d413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cbfff700 1 -- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 msgr2=0x7fc6d4138310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cbfff700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 0x7fc6d4138310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00c530 tx=0x7fc6bc007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc005d40 con 0x7fc6d4132d00 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: dump: 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.185 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 5 monclient: authenticate success, global_id 4476 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6bc0070d0 con 0x7fc6d4132d00 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:22.186 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 10 monclient: _renew_subs 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc6bc007490 con 0x7fc6d4132d00 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6d4133510 con 0x7fc6d4132d00 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6cb7fe700 10 monclient: dump: 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:22.187 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d27fc700 4 set_mon_vals no callback set 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 10 monclient: _renew_subs 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.174+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6d412e860 con 0x7fc6d4132d00 2023-09-30T00:46:22.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6d9dd6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6d9dd6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6d9dd6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc6bc01c440 con 0x7fc6d4132d00 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6d41330d0 con 0x7fc6d4132d00 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6cb7fe700 1 --2- 172.21.15.6:0/4220375850 >> v2:172.21.15.22:6812/142603 conn(0x7fc6b40357a0 0x7fc6b4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc6bc046e30 con 0x7fc6d4132d00 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6d37fe700 1 --2- 172.21.15.6:0/4220375850 >> v2:172.21.15.22:6812/142603 conn(0x7fc6b40357a0 0x7fc6b4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:22.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.175+0000 7fc6d37fe700 10 monclient: get_auth_request con 0x7fc6b40357a0 auth_method 0 2023-09-30T00:46:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.178+0000 7fc6d37fe700 1 --2- 172.21.15.6:0/4220375850 >> v2:172.21.15.22:6812/142603 conn(0x7fc6b40357a0 0x7fc6b4037c50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4007fd0 tx=0x7fc6c4007f60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.180+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc6bc018850 con 0x7fc6d4132d00 2023-09-30T00:46:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.180+0000 7fc6cb7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:22.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.180+0000 7fc6cb7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.424+0000 7fc6d9dd6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.424+0000 7fc6d9dd6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:22.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.424+0000 7fc6d9dd6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:22.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.424+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc6d4072b00 con 0x7fc6d4132d00 2023-09-30T00:46:22.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.425+0000 7fc6cb7fe700 1 -- 172.21.15.6:0/4220375850 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc6bc017020 con 0x7fc6d4132d00 2023-09-30T00:46:22.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.425+0000 7fc6cb7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:22.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.425+0000 7fc6cb7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:22.425 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:22.425 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:22.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 >> v2:172.21.15.22:6812/142603 conn(0x7fc6b40357a0 msgr2=0x7fc6b4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:22.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/4220375850 >> v2:172.21.15.22:6812/142603 conn(0x7fc6b40357a0 0x7fc6b4037c50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc6c4007fd0 tx=0x7fc6c4007f60 comp rx=0 tx=0).stop 2023-09-30T00:46:22.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 10 monclient: shutdown 2023-09-30T00:46:22.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:22.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 msgr2=0x7fc6d4138310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 0x7fc6d4138310 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc6bc00c530 tx=0x7fc6bc007f90 comp rx=0 tx=0).stop 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 shutdown_connections 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/4220375850 >> v2:172.21.15.22:6812/142603 conn(0x7fc6b40357a0 0x7fc6b4037c50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc6d4138d40 0x7fc6d413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc6d4098670 0x7fc6d4141600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 --2- 172.21.15.6:0/4220375850 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc6d4132d00 0x7fc6d4138310 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 >> 172.21.15.6:0/4220375850 conn(0x7fc6d40a5480 msgr2=0x7fc6d41305a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 shutdown_connections 2023-09-30T00:46:22.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:22.427+0000 7fc6d9dd6700 1 -- 172.21.15.6:0/4220375850 wait complete. 2023-09-30T00:46:25.441 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:25.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.580+0000 7f5eaf8be700 10 monclient: get_monmap_and_config 2023-09-30T00:46:25.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.580+0000 7f5eaf8be700 10 monclient: build_initial_monmap 2023-09-30T00:46:25.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.580+0000 7f5eaf8be700 10 monclient: monmap: 2023-09-30T00:46:25.580 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:25.580 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:25.580 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:25.580737+0000 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:25.580737+0000 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:25.581 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:25.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 1 Processor -- start 2023-09-30T00:46:25.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 1 -- start start 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient: init 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5ea809e720 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5ea809eaf0 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5ea809eec0 0x7f5ea813ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient(hunting): picked mon.noname-c con 0x7f5ea809eec0 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea8069490 con 0x7f5ea809eaf0 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ea8069120 con 0x7f5ea809e720 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ea809f2e0 con 0x7f5ea809eec0 2023-09-30T00:46:25.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.581+0000 7f5eaf8be700 10 monclient(hunting): authenticate will time out at 1381.878894s 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eace59700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5ea809eaf0 0x7f5ea81375e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:57712/0 (socket says 172.21.15.6:57712) 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eace59700 1 -- 172.21.15.6:0/1958586291 learned_addr learned my addr 172.21.15.6:0/1958586291 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 1 --2- 172.21.15.6:0/1958586291 >> v2:172.21.15.22:3300/0 conn(0x7f5ea809eec0 0x7f5ea813ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): get_auth_request con 0x7f5ea809eec0 auth_method 0 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:25.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient(hunting): handle_auth_done global_id 4495 payload 274 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 1 -- 172.21.15.6:0/1958586291 >> v1:172.21.15.22:6789/0 conn(0x7f5ea809e720 legacy=0x7f5ea809dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 1 -- 172.21.15.6:0/1958586291 >> v1:172.21.15.6:6789/0 conn(0x7f5ea809eaf0 legacy=0x7f5ea81375e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient: _finish_hunting 0 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 1 monclient: found mon.noname-c 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:46:25.586 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.582+0000 7f5eade5b700 1 -- 172.21.15.6:0/1958586291 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea8069750 con 0x7f5ea809eec0 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.584+0000 7f5eade5b700 1 -- 172.21.15.6:0/1958586291 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809eec0 msgr2=0x7f5ea813ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.584+0000 7f5eade5b700 1 --2- 172.21.15.6:0/1958586291 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809eec0 0x7f5ea813ad10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009e60 tx=0x7f5ea0009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85ff5cac6876c87 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 1 -- 172.21.15.6:0/1958586291 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000e070 con 0x7f5ea809eec0 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: dump: 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:25.592 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: _finish_auth 0 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 1 -- 172.21.15.6:0/1958586291 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ea0012430 con 0x7f5ea809eec0 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:25.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 1 -- 172.21.15.6:0/1958586291 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ea000bea0 con 0x7f5ea809eec0 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5e9ffff700 10 monclient: dump: 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:25.594 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5eaf8be700 5 monclient: authenticate success, global_id 4495 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5eaf8be700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.585+0000 7f5eaf8be700 4 set_mon_vals no callback set 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 10 monclient: shutdown 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 -- 172.21.15.6:0/1958586291 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809eec0 msgr2=0x7f5ea813ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/1958586291 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809eec0 0x7f5ea813ad10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5ea0009e60 tx=0x7f5ea0009780 comp rx=0 tx=0).stop 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 -- 172.21.15.6:0/1958586291 shutdown_connections 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/1958586291 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809eec0 0x7f5ea813ad10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 -- 172.21.15.6:0/1958586291 >> 172.21.15.6:0/1958586291 conn(0x7f5ea809fdb0 msgr2=0x7f5ea80a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:25.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 -- 172.21.15.6:0/1958586291 shutdown_connections 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 1 -- 172.21.15.6:0/1958586291 wait complete. 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 10 monclient: build_initial_monmap 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.586+0000 7f5eaf8be700 10 monclient: monmap: 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:25.596 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 Processor -- start 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 -- start start 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient: init 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809e720 0x7f5ea81344c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): picked mon.noname-b con 0x7f5ea809e720 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 0x7f5ea8141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): picked mon.noname-a con 0x7f5ea809eaf0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5ea809eec0 0x7f5ea8141d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): picked mon.noname-c con 0x7f5ea809eec0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:25.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea806a4e0 con 0x7f5ea809eaf0 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81218c0 con 0x7f5ea809e720 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea81215e0 con 0x7f5ea809eec0 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eaf8be700 10 monclient(hunting): authenticate will time out at 1381.884756s 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 0x7f5ea8141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 0x7f5ea8141830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:33928/0 (socket says 172.21.15.6:33928) 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 1 -- 172.21.15.6:0/4132184119 learned_addr learned my addr 172.21.15.6:0/4132184119 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:25.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 10 monclient(hunting): get_auth_request con 0x7f5ea809eaf0 auth_method 0 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eade5b700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5ea809eec0 0x7f5ea8141d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eade5b700 10 monclient(hunting): get_auth_request con 0x7f5ea809eec0 auth_method 0 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eade5b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eade5b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eade5b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5ead65a700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809e720 0x7f5ea81344c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.587+0000 7f5eace59700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:25.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): get_auth_request con 0x7f5ea809e720 auth_method 0 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eade5b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eade5b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eade5b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5ead65a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 10 monclient(hunting): handle_auth_done global_id 4479 payload 824 2023-09-30T00:46:25.600 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 1 -- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5ea809eec0 msgr2=0x7f5ea8141d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5ea809eec0 0x7f5ea8141d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 1 -- 172.21.15.6:0/4132184119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809e720 msgr2=0x7f5ea81344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809e720 0x7f5ea81344c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 10 monclient: _finish_hunting 0 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 1 monclient: found mon.noname-a 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eace59700 1 -- 172.21.15.6:0/4132184119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea0009a20 con 0x7f5ea809eaf0 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.588+0000 7f5eade5b700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5ea809eec0 0x7f5ea8141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eace59700 1 -- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 msgr2=0x7f5ea8141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eace59700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 0x7f5ea8141830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c5e0 tx=0x7f5e98004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:25.601 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98005d40 con 0x7f5ea809eaf0 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: dump: 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:25.602 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: _finish_auth 0 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e980070d0 con 0x7f5ea809eaf0 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 5 monclient: authenticate success, global_id 4479 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 10 monclient: _renew_subs 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea811fad0 con 0x7f5ea809eaf0 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e98007490 con 0x7f5ea809eaf0 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:25.603 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9d7fa700 10 monclient: dump: 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 10 monclient: _renew_subs 2023-09-30T00:46:25.604 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5e9f7fe700 4 set_mon_vals no callback set 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.590+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea8134bc0 con 0x7f5ea809eaf0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5e9801b3b0 con 0x7f5ea809eaf0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5eaf8be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5eaf8be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5eaf8be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5e9d7fa700 1 --2- 172.21.15.6:0/4132184119 >> v2:172.21.15.22:6812/142603 conn(0x7f5e8c0357a0 0x7f5e8c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea8072b00 con 0x7f5ea809eaf0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.591+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5e98046dd0 con 0x7f5ea809eaf0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.592+0000 7f5ead65a700 1 --2- 172.21.15.6:0/4132184119 >> v2:172.21.15.22:6812/142603 conn(0x7f5e8c0357a0 0x7f5e8c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:25.605 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.592+0000 7f5ead65a700 10 monclient: get_auth_request con 0x7f5e8c0357a0 auth_method 0 2023-09-30T00:46:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.594+0000 7f5ead65a700 1 --2- 172.21.15.6:0/4132184119 >> v2:172.21.15.22:6812/142603 conn(0x7f5e8c0357a0 0x7f5e8c037c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4003a70 tx=0x7f5ea4007ef0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.596+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e98017020 con 0x7f5ea809eaf0 2023-09-30T00:46:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.596+0000 7f5e9d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:25.606 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.596+0000 7f5e9d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:25.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.845+0000 7f5eaf8be700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:25.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.845+0000 7f5eaf8be700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:25.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.845+0000 7f5eaf8be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:25.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.845+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5ea8121250 con 0x7f5ea809eaf0 2023-09-30T00:46:25.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.846+0000 7f5e9d7fa700 1 -- 172.21.15.6:0/4132184119 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5e9801a360 con 0x7f5ea809eaf0 2023-09-30T00:46:25.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.846+0000 7f5e9d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:25.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.846+0000 7f5e9d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:25.846 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:25.846 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 >> v2:172.21.15.22:6812/142603 conn(0x7f5e8c0357a0 msgr2=0x7f5e8c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/4132184119 >> v2:172.21.15.22:6812/142603 conn(0x7f5e8c0357a0 0x7f5e8c037c50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5ea4003a70 tx=0x7f5ea4007ef0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 10 monclient: shutdown 2023-09-30T00:46:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 msgr2=0x7f5ea8141830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:25.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 0x7f5ea8141830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5e9800c5e0 tx=0x7f5e98004a80 comp rx=0 tx=0).stop 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 shutdown_connections 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/4132184119 >> v2:172.21.15.22:6812/142603 conn(0x7f5e8c0357a0 0x7f5e8c037c50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5ea809eec0 0x7f5ea8141d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5ea809e720 0x7f5ea81344c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 --2- 172.21.15.6:0/4132184119 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5ea809eaf0 0x7f5ea8141830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 >> 172.21.15.6:0/4132184119 conn(0x7f5ea809fdb0 msgr2=0x7f5ea809c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 shutdown_connections 2023-09-30T00:46:25.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:25.848+0000 7f5eaf8be700 1 -- 172.21.15.6:0/4132184119 wait complete. 2023-09-30T00:46:28.863 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.000+0000 7fd59ca23700 10 monclient: get_monmap_and_config 2023-09-30T00:46:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.000+0000 7fd59ca23700 10 monclient: build_initial_monmap 2023-09-30T00:46:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.000+0000 7fd59ca23700 10 monclient: monmap: 2023-09-30T00:46:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:29.001 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:29.001753+0000 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:29.001753+0000 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:29.002 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:29.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 1 Processor -- start 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 1 -- start start 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient: init 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient: _add_conns ranks=[4,0,1,2,3] 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fd598098670 0x7fd598132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient(hunting): picked mon.noname-c con 0x7fd598098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd598098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd598098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5980a1000 con 0x7fd598098a40 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59812fc00 con 0x7fd598098e90 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:29.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.001+0000 7fd59ca23700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5980a0a50 con 0x7fd598098670 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59ca23700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59ca23700 10 monclient(hunting): authenticate will time out at 1385.300012s 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd595d9b700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd598098a40 0x7fd59812ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45436/0 (socket says 172.21.15.6:45436) 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd595d9b700 1 -- 172.21.15.6:0/173060857 learned_addr learned my addr 172.21.15.6:0/173060857 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 1 --2- 172.21.15.6:0/173060857 >> v2:172.21.15.22:3300/0 conn(0x7fd598098670 0x7fd598132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): get_auth_request con 0x7fd598098670 auth_method 0 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:29.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient(hunting): handle_auth_done global_id 4501 payload 274 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 1 -- 172.21.15.6:0/173060857 >> v1:172.21.15.22:6789/0 conn(0x7fd598098e90 legacy=0x7fd59812f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 1 -- 172.21.15.6:0/173060857 >> v1:172.21.15.6:6789/0 conn(0x7fd598098a40 legacy=0x7fd59812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient: _finish_hunting 0 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 1 monclient: found mon.noname-c 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:46:29.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.002+0000 7fd59659c700 1 -- 172.21.15.6:0/173060857 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5980a0730 con 0x7fd598098670 2023-09-30T00:46:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.004+0000 7fd59659c700 1 -- 172.21.15.6:0/173060857 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098670 msgr2=0x7fd598132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.004+0000 7fd59659c700 1 --2- 172.21.15.6:0/173060857 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098670 0x7fd598132d00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5880098e0 tx=0x7fd5880091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bffea08dcbc5e52d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 1 -- 172.21.15.6:0/173060857 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58800b9f0 con 0x7fd598098670 2023-09-30T00:46:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:29.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: dump: 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: _finish_auth 0 2023-09-30T00:46:29.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 1 -- 172.21.15.6:0/173060857 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58800bb50 con 0x7fd598098670 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 5 monclient: authenticate success, global_id 4501 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 1 -- 172.21.15.6:0/173060857 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd588011420 con 0x7fd598098670 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 4 set_mon_vals no callback set 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59559a700 10 monclient: dump: 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:29.015 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 10 monclient: shutdown 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 -- 172.21.15.6:0/173060857 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098670 msgr2=0x7fd598132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:29.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 --2- 172.21.15.6:0/173060857 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098670 0x7fd598132d00 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd5880098e0 tx=0x7fd5880091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 -- 172.21.15.6:0/173060857 shutdown_connections 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 --2- 172.21.15.6:0/173060857 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098670 0x7fd598132d00 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 -- 172.21.15.6:0/173060857 >> 172.21.15.6:0/173060857 conn(0x7fd5980a5480 msgr2=0x7fd5980a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 -- 172.21.15.6:0/173060857 shutdown_connections 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.005+0000 7fd59ca23700 1 -- 172.21.15.6:0/173060857 wait complete. 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.006+0000 7fd59ca23700 10 monclient: build_initial_monmap 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.006+0000 7fd59ca23700 10 monclient: monmap: 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:29.017 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.006+0000 7fd59ca23700 1 Processor -- start 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.006+0000 7fd59ca23700 1 -- start start 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.006+0000 7fd59ca23700 10 monclient: init 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 0x7fd59812c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:29.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): picked mon.noname-a con 0x7fd598098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098a40 0x7fd5981417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): picked mon.noname-b con 0x7fd598098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd598098e90 0x7fd59809cac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): picked mon.noname-c con 0x7fd598098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5980a3af0 con 0x7fd598098670 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59811c8f0 con 0x7fd598098a40 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59809f210 con 0x7fd598098e90 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:29.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59ca23700 10 monclient(hunting): authenticate will time out at 1385.305357s 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 0x7fd59812c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 0x7fd59812c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41006/0 (socket says 172.21.15.6:41006) 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 -- 172.21.15.6:0/2121144879 learned_addr learned my addr 172.21.15.6:0/2121144879 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): get_auth_request con 0x7fd598098670 auth_method 0 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd598098e90 0x7fd59809cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): get_auth_request con 0x7fd598098e90 auth_method 0 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:29.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098a40 0x7fd5981417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): get_auth_request con 0x7fd598098a40 auth_method 0 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:29.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd595d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient(hunting): handle_auth_done global_id 4482 payload 824 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 -- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd598098e90 msgr2=0x7fd59809cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd598098e90 0x7fd59809cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 -- 172.21.15.6:0/2121144879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098a40 msgr2=0x7fd5981417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098a40 0x7fd5981417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient: _finish_hunting 0 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 monclient: found mon.noname-a 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:29.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd59659c700 1 -- 172.21.15.6:0/2121144879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd588009420 con 0x7fd598098670 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.007+0000 7fd596d9d700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd598098e90 0x7fd59809cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59659c700 1 -- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 msgr2=0x7fd59812c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59659c700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 0x7fd59812c150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd59806a6b0 tx=0x7fd588016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58800bed0 con 0x7fd598098670 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: dump: 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:29.023 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd588020070 con 0x7fd598098670 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 5 monclient: authenticate success, global_id 4482 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 10 monclient: _renew_subs 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59811c2a0 con 0x7fd598098670 2023-09-30T00:46:29.024 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd58801b680 con 0x7fd598098670 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd5867fc700 10 monclient: dump: 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:29.025 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd594d99700 4 set_mon_vals no callback set 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 10 monclient: _renew_subs 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.010+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59811c630 con 0x7fd598098670 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd588022030 con 0x7fd598098670 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd5867fc700 1 --2- 172.21.15.6:0/2121144879 >> v2:172.21.15.22:6812/142603 conn(0x7fd5780357a0 0x7fd578037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd59ca23700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd59ca23700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd59ca23700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:29.026 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd588049730 con 0x7fd598098670 2023-09-30T00:46:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd598072b00 con 0x7fd598098670 2023-09-30T00:46:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd595d9b700 1 --2- 172.21.15.6:0/2121144879 >> v2:172.21.15.22:6812/142603 conn(0x7fd5780357a0 0x7fd578037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.011+0000 7fd595d9b700 10 monclient: get_auth_request con 0x7fd5780357a0 auth_method 0 2023-09-30T00:46:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.014+0000 7fd595d9b700 1 --2- 172.21.15.6:0/2121144879 >> v2:172.21.15.22:6812/142603 conn(0x7fd5780357a0 0x7fd578037c50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5980a42d0 tx=0x7fd58c007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.016+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd58801ba30 con 0x7fd598098670 2023-09-30T00:46:29.027 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.016+0000 7fd5867fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:29.028 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.016+0000 7fd5867fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd59ca23700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd59ca23700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd59ca23700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd598072b00 con 0x7fd598098670 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd5867fc700 1 -- 172.21.15.6:0/2121144879 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd58801a3d0 con 0x7fd598098670 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd5867fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:29.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.266+0000 7fd5867fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:29.268 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:29.268 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 >> v2:172.21.15.22:6812/142603 conn(0x7fd5780357a0 msgr2=0x7fd578037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 --2- 172.21.15.6:0/2121144879 >> v2:172.21.15.22:6812/142603 conn(0x7fd5780357a0 0x7fd578037c50 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fd5980a42d0 tx=0x7fd58c007f90 comp rx=0 tx=0).stop 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 10 monclient: shutdown 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 msgr2=0x7fd59812c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 0x7fd59812c150 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fd59806a6b0 tx=0x7fd588016000 comp rx=0 tx=0).stop 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 shutdown_connections 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 --2- 172.21.15.6:0/2121144879 >> v2:172.21.15.22:6812/142603 conn(0x7fd5780357a0 0x7fd578037c50 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd598098e90 0x7fd59809cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd598098670 0x7fd59812c150 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.268+0000 7fd59ca23700 1 --2- 172.21.15.6:0/2121144879 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd598098a40 0x7fd5981417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:29.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.269+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 >> 172.21.15.6:0/2121144879 conn(0x7fd5980a5480 msgr2=0x7fd59812f0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:29.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.269+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 shutdown_connections 2023-09-30T00:46:29.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:29.269+0000 7fd59ca23700 1 -- 172.21.15.6:0/2121144879 wait complete. 2023-09-30T00:46:32.283 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:32.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.421+0000 7fd45de09700 10 monclient: get_monmap_and_config 2023-09-30T00:46:32.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.421+0000 7fd45de09700 10 monclient: build_initial_monmap 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.421+0000 7fd45de09700 10 monclient: monmap: 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:32.423090+0000 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:32.423090+0000 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:32.423 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:32.424 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.422+0000 7fd45de09700 1 Processor -- start 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.422+0000 7fd45de09700 1 -- start start 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.422+0000 7fd45de09700 10 monclient: init 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd458132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): picked mon.noname-b con 0x7fd458098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fd458098a40 0x7fd45812ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:32.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): picked mon.noname-c con 0x7fd458098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd458098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4580a1000 con 0x7fd458098e90 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4580a0a50 con 0x7fd458098a40 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580a0730 con 0x7fd458098670 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd45de09700 10 monclient(hunting): authenticate will time out at 1388.721195s 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd457fff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd458098e90 0x7fd45812f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45452/0 (socket says 172.21.15.6:45452) 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd457fff700 1 -- 172.21.15.6:0/3675483717 learned_addr learned my addr 172.21.15.6:0/3675483717 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 1 --2- 172.21.15.6:0/3675483717 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd458132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:32.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): get_auth_request con 0x7fd458098670 auth_method 0 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 1 --2- 172.21.15.6:0/3675483717 >> v2:172.21.15.22:3300/0 conn(0x7fd458098a40 0x7fd45812ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): get_auth_request con 0x7fd458098a40 auth_method 0 2023-09-30T00:46:32.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd456ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient(hunting): handle_auth_done global_id 4487 payload 274 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 1 -- 172.21.15.6:0/3675483717 >> v2:172.21.15.22:3300/0 conn(0x7fd458098a40 msgr2=0x7fd45812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 1 --2- 172.21.15.6:0/3675483717 >> v2:172.21.15.22:3300/0 conn(0x7fd458098a40 0x7fd45812ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 1 -- 172.21.15.6:0/3675483717 >> v1:172.21.15.6:6789/0 conn(0x7fd458098e90 legacy=0x7fd45812f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 1 monclient: found mon.noname-b 2023-09-30T00:46:32.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:32.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:32.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.423+0000 7fd4577fe700 1 -- 172.21.15.6:0/3675483717 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd458068df0 con 0x7fd458098670 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.425+0000 7fd4577fe700 1 -- 172.21.15.6:0/3675483717 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 msgr2=0x7fd458132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.425+0000 7fd4577fe700 1 --2- 172.21.15.6:0/3675483717 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd458132d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd4480098e0 tx=0x7fd4480091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f16eac6b6c62141 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 1 -- 172.21.15.6:0/3675483717 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448011840 con 0x7fd458098670 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: dump: 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.436 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 1 -- 172.21.15.6:0/3675483717 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4480119a0 con 0x7fd458098670 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 5 monclient: authenticate success, global_id 4487 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:32.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 1 -- 172.21.15.6:0/3675483717 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448011d40 con 0x7fd458098670 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd4567fc700 10 monclient: dump: 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:32.438 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 4 set_mon_vals no callback set 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 10 monclient: shutdown 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 -- 172.21.15.6:0/3675483717 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 msgr2=0x7fd458132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 --2- 172.21.15.6:0/3675483717 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd458132d00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fd4480098e0 tx=0x7fd4480091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 -- 172.21.15.6:0/3675483717 shutdown_connections 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 --2- 172.21.15.6:0/3675483717 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd458132d00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 --2- 172.21.15.6:0/3675483717 >> v2:172.21.15.22:3300/0 conn(0x7fd458098a40 0x7fd45812ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 -- 172.21.15.6:0/3675483717 >> 172.21.15.6:0/3675483717 conn(0x7fd4580a5480 msgr2=0x7fd4580a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:32.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 -- 172.21.15.6:0/3675483717 shutdown_connections 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.426+0000 7fd45de09700 1 -- 172.21.15.6:0/3675483717 wait complete. 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.427+0000 7fd45de09700 10 monclient: build_initial_monmap 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.427+0000 7fd45de09700 10 monclient: monmap: 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:32.440 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.427+0000 7fd45de09700 1 Processor -- start 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 -- start start 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient: init 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd4581419a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): picked mon.noname-c con 0x7fd458098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 0x7fd45812f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): picked mon.noname-a con 0x7fd458098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd458098e90 0x7fd45809c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): picked mon.noname-b con 0x7fd458098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:32.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580a1500 con 0x7fd458098a40 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4580a3e00 con 0x7fd458098e90 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd45809ef40 con 0x7fd458098670 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd45de09700 10 monclient(hunting): authenticate will time out at 1388.726569s 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 0x7fd45812f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd4581419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 0x7fd45812f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41022/0 (socket says 172.21.15.6:41022) 2023-09-30T00:46:32.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 1 -- 172.21.15.6:0/653527694 learned_addr learned my addr 172.21.15.6:0/653527694 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd4581419a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:45706/0 (socket says 172.21.15.6:45706) 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): get_auth_request con 0x7fd458098a40 auth_method 0 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): get_auth_request con 0x7fd458098670 auth_method 0 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:32.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd456ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd457fff700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd458098e90 0x7fd45809c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd457fff700 10 monclient(hunting): get_auth_request con 0x7fd458098e90 auth_method 0 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd457fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd457fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd457fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.428+0000 7fd4577fe700 10 monclient(hunting): handle_auth_done global_id 4490 payload 824 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 1 -- 172.21.15.6:0/653527694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd458098e90 msgr2=0x7fd45809c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd458098e90 0x7fd45809c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 1 -- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 msgr2=0x7fd45812f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:32.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 0x7fd45812f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 1 monclient: found mon.noname-c 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd4577fe700 1 -- 172.21.15.6:0/653527694 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd448009420 con 0x7fd458098670 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.429+0000 7fd456ffd700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 0x7fd45812f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd4577fe700 1 -- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 msgr2=0x7fd4581419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd4577fe700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd4581419a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4580a1df0 tx=0x7fd448016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd448000c40 con 0x7fd458098670 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:32.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: dump: 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:32.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd448020070 con 0x7fd458098670 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 5 monclient: authenticate success, global_id 4490 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 10 monclient: _renew_subs 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd45812d390 con 0x7fd458098670 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44801b450 con 0x7fd458098670 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd43f7fe700 10 monclient: dump: 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:32.447 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd455ffb700 4 set_mon_vals no callback set 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 10 monclient: _renew_subs 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.431+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4581427d0 con 0x7fd458098670 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd448022030 con 0x7fd458098670 2023-09-30T00:46:32.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd45de09700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd45de09700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd45de09700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd45812da70 con 0x7fd458098670 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd43f7fe700 1 --2- 172.21.15.6:0/653527694 >> v2:172.21.15.22:6812/142603 conn(0x7fd4440357a0 0x7fd444037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd448049510 con 0x7fd458098670 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.432+0000 7fd456ffd700 1 --2- 172.21.15.6:0/653527694 >> v2:172.21.15.22:6812/142603 conn(0x7fd4440357a0 0x7fd444037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.433+0000 7fd456ffd700 10 monclient: get_auth_request con 0x7fd4440357a0 auth_method 0 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.435+0000 7fd456ffd700 1 --2- 172.21.15.6:0/653527694 >> v2:172.21.15.22:6812/142603 conn(0x7fd4440357a0 0x7fd444037c50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009fd0 tx=0x7fd44c0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:32.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.437+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd44801c450 con 0x7fd458098670 2023-09-30T00:46:32.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.437+0000 7fd43f7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:32.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.437+0000 7fd43f7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.684+0000 7fd45de09700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.684+0000 7fd45de09700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:32.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.684+0000 7fd45de09700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.684+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd45812da70 con 0x7fd458098670 2023-09-30T00:46:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.685+0000 7fd43f7fe700 1 -- 172.21.15.6:0/653527694 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd448010e60 con 0x7fd458098670 2023-09-30T00:46:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.685+0000 7fd43f7fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:32.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.685+0000 7fd43f7fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:32.686 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:32.686 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 >> v2:172.21.15.22:6812/142603 conn(0x7fd4440357a0 msgr2=0x7fd444037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 --2- 172.21.15.6:0/653527694 >> v2:172.21.15.22:6812/142603 conn(0x7fd4440357a0 0x7fd444037c50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fd44c009fd0 tx=0x7fd44c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 10 monclient: shutdown 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 msgr2=0x7fd4581419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd4581419a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fd4580a1df0 tx=0x7fd448016000 comp rx=0 tx=0).stop 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 shutdown_connections 2023-09-30T00:46:32.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 --2- 172.21.15.6:0/653527694 >> v2:172.21.15.22:6812/142603 conn(0x7fd4440357a0 0x7fd444037c50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd458098e90 0x7fd45809c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd458098670 0x7fd4581419a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 --2- 172.21.15.6:0/653527694 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd458098a40 0x7fd45812f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 >> 172.21.15.6:0/653527694 conn(0x7fd4580a5480 msgr2=0x7fd4580a4010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 shutdown_connections 2023-09-30T00:46:32.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:32.687+0000 7fd45de09700 1 -- 172.21.15.6:0/653527694 wait complete. 2023-09-30T00:46:35.702 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.840+0000 7f4e8a31f700 10 monclient: get_monmap_and_config 2023-09-30T00:46:35.841 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.840+0000 7f4e8a31f700 10 monclient: build_initial_monmap 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.840+0000 7f4e8a31f700 10 monclient: monmap: 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:35.842113+0000 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:35.842113+0000 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:35.842 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:35.843 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:35.843 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.841+0000 7f4e8a31f700 1 Processor -- start 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.841+0000 7f4e8a31f700 1 -- start start 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.841+0000 7f4e8a31f700 10 monclient: init 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient: _add_conns ranks=[3,1,4,0,2] 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4e84098670 0x7f4e84132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): picked mon.noname-a con 0x7f4e84098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4e84098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4e84098e90 0x7f4e8412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:35.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): picked mon.noname-c con 0x7f4e84098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e840a1000 con 0x7f4e84098a40 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840a0a50 con 0x7f4e84098670 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e840a0730 con 0x7f4e84098e90 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e8a31f700 10 monclient(hunting): authenticate will time out at 1392.140274s 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4e84098670 0x7f4e84132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4e84098670 0x7f4e84132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41032/0 (socket says 172.21.15.6:41032) 2023-09-30T00:46:35.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 -- 172.21.15.6:0/3123166637 learned_addr learned my addr 172.21.15.6:0/3123166637 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): get_auth_request con 0x7f4e84098670 auth_method 0 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e888bc700 1 --2- 172.21.15.6:0/3123166637 >> v2:172.21.15.22:3300/0 conn(0x7f4e84098e90 0x7f4e8412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient(hunting): handle_auth_done global_id 4488 payload 274 2023-09-30T00:46:35.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 -- 172.21.15.6:0/3123166637 >> v2:172.21.15.22:3300/0 conn(0x7f4e84098e90 msgr2=0x7f4e8412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 --2- 172.21.15.6:0/3123166637 >> v2:172.21.15.22:3300/0 conn(0x7f4e84098e90 0x7f4e8412f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 -- 172.21.15.6:0/3123166637 >> v1:172.21.15.22:6789/0 conn(0x7f4e84098a40 legacy=0x7f4e8412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 monclient: found mon.noname-a 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e83fff700 1 -- 172.21.15.6:0/3123166637 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e84068df0 con 0x7f4e84098670 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e888bc700 10 monclient: get_auth_request con 0x7f4e84098e90 auth_method 0 2023-09-30T00:46:35.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.842+0000 7f4e888bc700 1 --2- 172.21.15.6:0/3123166637 >> v2:172.21.15.22:3300/0 conn(0x7f4e84098e90 0x7f4e8412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:46:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e83fff700 1 -- 172.21.15.6:0/3123166637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098670 msgr2=0x7f4e84132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e83fff700 1 --2- 172.21.15.6:0/3123166637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098670 0x7f4e84132d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e840a3330 tx=0x7f4e74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be3d21d4c5fe96e6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 1 -- 172.21.15.6:0/3123166637 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7400ba30 con 0x7f4e84098670 2023-09-30T00:46:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:35.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: dump: 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:35.855 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 1 -- 172.21.15.6:0/3123166637 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e7400bb90 con 0x7f4e84098670 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 5 monclient: authenticate success, global_id 4488 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 1 -- 172.21.15.6:0/3123166637 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e74011420 con 0x7f4e84098670 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:35.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 4 set_mon_vals no callback set 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e82ffd700 10 monclient: dump: 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:35.857 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 10 monclient: shutdown 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3123166637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098670 msgr2=0x7f4e84132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3123166637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098670 0x7f4e84132d00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f4e840a3330 tx=0x7f4e74009250 comp rx=0 tx=0).stop 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3123166637 shutdown_connections 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3123166637 >> v2:172.21.15.22:3300/0 conn(0x7f4e84098e90 0x7f4e8412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3123166637 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098670 0x7f4e84132d00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3123166637 >> 172.21.15.6:0/3123166637 conn(0x7f4e840a5480 msgr2=0x7f4e840a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3123166637 shutdown_connections 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.845+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3123166637 wait complete. 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.846+0000 7f4e8a31f700 10 monclient: build_initial_monmap 2023-09-30T00:46:35.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.846+0000 7f4e8a31f700 10 monclient: monmap: 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.846+0000 7f4e8a31f700 1 Processor -- start 2023-09-30T00:46:35.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 -- start start 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient: init 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4e84098670 0x7f4e8412d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): picked mon.noname-b con 0x7f4e84098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 0x7f4e8412d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): picked mon.noname-c con 0x7f4e84098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 0x7f4e8409d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): picked mon.noname-a con 0x7f4e84098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:35.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e840a1500 con 0x7f4e84098e90 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8409d220 con 0x7f4e84098670 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e8412ecd0 con 0x7f4e84098a40 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e8a31f700 10 monclient(hunting): authenticate will time out at 1392.145562s 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 0x7f4e8409d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 0x7f4e8409d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:41038/0 (socket says 172.21.15.6:41038) 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 0x7f4e8412d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 1 -- 172.21.15.6:0/3142005146 learned_addr learned my addr 172.21.15.6:0/3142005146 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): get_auth_request con 0x7f4e84098e90 auth_method 0 2023-09-30T00:46:35.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): get_auth_request con 0x7f4e84098a40 auth_method 0 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4e84098670 0x7f4e8412d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e888bc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:35.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e837fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): get_auth_request con 0x7f4e84098670 auth_method 0 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.847+0000 7f4e83fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 10 monclient(hunting): handle_auth_done global_id 4493 payload 824 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 1 -- 172.21.15.6:0/3142005146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4e84098670 msgr2=0x7f4e8412d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4e84098670 0x7f4e8412d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:35.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 1 -- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 msgr2=0x7f4e8409d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 0x7f4e8409d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 1 monclient: found mon.noname-c 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e837fe700 1 -- 172.21.15.6:0/3142005146 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e740094f0 con 0x7f4e84098a40 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.848+0000 7f4e888bc700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 0x7f4e8409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e837fe700 1 -- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 msgr2=0x7f4e8412d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e837fe700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 0x7f4e8412d8d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c4e0 tx=0x7f4e78007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e7800ce90 con 0x7f4e84098a40 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:35.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: dump: 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: _finish_auth 0 2023-09-30T00:46:35.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e780117a0 con 0x7f4e84098a40 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 5 monclient: authenticate success, global_id 4493 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 10 monclient: _renew_subs 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e8409cc60 con 0x7f4e84098a40 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e78011b40 con 0x7f4e84098a40 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e6bfff700 10 monclient: dump: 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:35.866 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e827fc700 4 set_mon_vals no callback set 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 10 monclient: _renew_subs 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.850+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e8412e750 con 0x7f4e84098a40 2023-09-30T00:46:35.867 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4e7801c460 con 0x7f4e84098a40 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e6bfff700 1 --2- 172.21.15.6:0/3142005146 >> v2:172.21.15.22:6812/142603 conn(0x7f4e700357a0 0x7f4e70037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4e78047540 con 0x7f4e84098a40 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e8a31f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e8a31f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e8a31f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e8409cea0 con 0x7f4e84098a40 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e83fff700 1 --2- 172.21.15.6:0/3142005146 >> v2:172.21.15.22:6812/142603 conn(0x7f4e700357a0 0x7f4e70037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:35.868 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.851+0000 7f4e83fff700 10 monclient: get_auth_request con 0x7f4e700357a0 auth_method 0 2023-09-30T00:46:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.854+0000 7f4e83fff700 1 --2- 172.21.15.6:0/3142005146 >> v2:172.21.15.22:6812/142603 conn(0x7f4e700357a0 0x7f4e70037c50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e840a3330 tx=0x7f4e74016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.856+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4e8409cea0 con 0x7f4e84098a40 2023-09-30T00:46:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.856+0000 7f4e6bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:35.869 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:35.856+0000 7f4e6bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.102+0000 7f4e8a31f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.102+0000 7f4e8a31f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.102+0000 7f4e8a31f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:36.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.102+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4e8409cea0 con 0x7f4e84098a40 2023-09-30T00:46:36.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.103+0000 7f4e6bfff700 1 -- 172.21.15.6:0/3142005146 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4e8409cea0 con 0x7f4e84098a40 2023-09-30T00:46:36.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.103+0000 7f4e6bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:36.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.103+0000 7f4e6bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:36.104 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:36.104 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:36.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 >> v2:172.21.15.22:6812/142603 conn(0x7f4e700357a0 msgr2=0x7f4e70037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:36.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3142005146 >> v2:172.21.15.22:6812/142603 conn(0x7f4e700357a0 0x7f4e70037c50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4e840a3330 tx=0x7f4e74016000 comp rx=0 tx=0).stop 2023-09-30T00:46:36.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 10 monclient: shutdown 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 msgr2=0x7f4e8412d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 0x7f4e8412d8d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4e7800c4e0 tx=0x7f4e78007ea0 comp rx=0 tx=0).stop 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 shutdown_connections 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3142005146 >> v2:172.21.15.22:6812/142603 conn(0x7f4e700357a0 0x7f4e70037c50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4e84098e90 0x7f4e8409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4e84098670 0x7f4e8412d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 --2- 172.21.15.6:0/3142005146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4e84098a40 0x7f4e8412d8d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 >> 172.21.15.6:0/3142005146 conn(0x7f4e840a5480 msgr2=0x7f4e840a3f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 shutdown_connections 2023-09-30T00:46:36.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:36.105+0000 7f4e8a31f700 1 -- 172.21.15.6:0/3142005146 wait complete. 2023-09-30T00:46:39.119 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:39.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.258+0000 7fa7fd2e6700 10 monclient: get_monmap_and_config 2023-09-30T00:46:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.258+0000 7fa7fd2e6700 10 monclient: build_initial_monmap 2023-09-30T00:46:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.259+0000 7fa7fd2e6700 10 monclient: monmap: 2023-09-30T00:46:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:39.260 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:39.260208+0000 2023-09-30T00:46:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:39.260208+0000 2023-09-30T00:46:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:39.261 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:39.262 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:39.262 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:39.262 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.259+0000 7fa7fd2e6700 1 Processor -- start 2023-09-30T00:46:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.259+0000 7fa7fd2e6700 1 -- start start 2023-09-30T00:46:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.259+0000 7fa7fd2e6700 10 monclient: init 2023-09-30T00:46:39.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa7f81323a0 0x7fa7f8132770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): picked mon.noname-c con 0x7fa7f81323a0 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa7f8098570 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7f81374e0 0x7fa7f81398c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): picked mon.noname-a con 0x7fa7f81374e0 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7f8096a80 con 0x7fa7f8098570 2023-09-30T00:46:39.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f8096d60 con 0x7fa7f81374e0 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7f80a1c30 con 0x7fa7f81323a0 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7fd2e6700 10 monclient(hunting): authenticate will time out at 1395.558267s 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7f81374e0 0x7fa7f81398c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa7f81374e0 0x7fa7f81398c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42178/0 (socket says 172.21.15.6:42178) 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 -- 172.21.15.6:0/688528941 learned_addr learned my addr 172.21.15.6:0/688528941 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:39.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): get_auth_request con 0x7fa7f81374e0 auth_method 0 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 1 --2- 172.21.15.6:0/688528941 >> v2:172.21.15.22:3300/0 conn(0x7fa7f81323a0 0x7fa7f8132770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): get_auth_request con 0x7fa7f81323a0 auth_method 0 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_done global_id 4494 payload 274 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 -- 172.21.15.6:0/688528941 >> v2:172.21.15.22:3300/0 conn(0x7fa7f81323a0 msgr2=0x7fa7f8132770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 --2- 172.21.15.6:0/688528941 >> v2:172.21.15.22:3300/0 conn(0x7fa7f81323a0 0x7fa7f8132770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 -- 172.21.15.6:0/688528941 >> v1:172.21.15.22:6789/0 conn(0x7fa7f8098570 legacy=0x7fa7f80989f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 monclient: found mon.noname-a 2023-09-30T00:46:39.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.260+0000 7fa7f77fe700 1 -- 172.21.15.6:0/688528941 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f80a0a20 con 0x7fa7f81374e0 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f77fe700 1 -- 172.21.15.6:0/688528941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 msgr2=0x7fa7f81398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f77fe700 1 --2- 172.21.15.6:0/688528941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f81398c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80a53e0 tx=0x7fa7f0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5dd9debf9e231f5 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 1 -- 172.21.15.6:0/688528941 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f000e070 con 0x7fa7f81374e0 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: dump: 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:39.276 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 1 -- 172.21.15.6:0/688528941 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f000bbf0 con 0x7fa7f81374e0 2023-09-30T00:46:39.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 5 monclient: authenticate success, global_id 4494 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 1 -- 172.21.15.6:0/688528941 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f00092e0 con 0x7fa7f81374e0 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 4 set_mon_vals no callback set 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7f5ffb700 10 monclient: dump: 2023-09-30T00:46:39.278 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 10 monclient: shutdown 2023-09-30T00:46:39.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/688528941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 msgr2=0x7fa7f81398c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/688528941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f81398c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa7f80a53e0 tx=0x7fa7f0009840 comp rx=0 tx=0).stop 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/688528941 shutdown_connections 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/688528941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f81398c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/688528941 >> v2:172.21.15.22:3300/0 conn(0x7fa7f81323a0 0x7fa7f8132770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.263+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/688528941 >> 172.21.15.6:0/688528941 conn(0x7fa7f80a5480 msgr2=0x7fa7f80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.264+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/688528941 shutdown_connections 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.264+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/688528941 wait complete. 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.264+0000 7fa7fd2e6700 10 monclient: build_initial_monmap 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.264+0000 7fa7fd2e6700 10 monclient: monmap: 2023-09-30T00:46:39.280 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:39.281 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.265+0000 7fa7fd2e6700 1 Processor -- start 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.265+0000 7fa7fd2e6700 1 -- start start 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.265+0000 7fa7fd2e6700 10 monclient: init 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7f8098570 0x7fa7f812f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): picked mon.noname-b con 0x7fa7f8098570 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa7f81323a0 0x7fa7f8129050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.282 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): picked mon.noname-c con 0x7fa7f81323a0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f8129590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): picked mon.noname-a con 0x7fa7f81374e0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f80a11b0 con 0x7fa7f81374e0 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f8139b70 con 0x7fa7f8098570 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7f81263a0 con 0x7fa7f81323a0 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7fd2e6700 10 monclient(hunting): authenticate will time out at 1395.564330s 2023-09-30T00:46:39.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f8129590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f8129590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42180/0 (socket says 172.21.15.6:42180) 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 -- 172.21.15.6:0/1131473519 learned_addr learned my addr 172.21.15.6:0/1131473519 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): get_auth_request con 0x7fa7f81374e0 auth_method 0 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa7f81323a0 0x7fa7f8129050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): get_auth_request con 0x7fa7f81323a0 auth_method 0 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7f8098570 0x7fa7f812f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): get_auth_request con 0x7fa7f8098570 auth_method 0 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:39.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient(hunting): handle_auth_done global_id 4497 payload 824 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 -- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa7f81323a0 msgr2=0x7fa7f8129050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa7f81323a0 0x7fa7f8129050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 -- 172.21.15.6:0/1131473519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7f8098570 msgr2=0x7fa7f812f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7f8098570 0x7fa7f812f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 monclient: found mon.noname-a 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f77fe700 1 -- 172.21.15.6:0/1131473519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7f0009ae0 con 0x7fa7f81374e0 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.266+0000 7fa7f67fc700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa7f81323a0 0x7fa7f8129050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7f77fe700 1 -- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 msgr2=0x7fa7f8129590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7f77fe700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f8129590 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000b6c0 tx=0x7fa7f0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f0012a60 con 0x7fa7f81374e0 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:39.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: dump: 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:39.288 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: _finish_auth 0 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 5 monclient: authenticate success, global_id 4497 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7f0012be0 con 0x7fa7f81374e0 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 10 monclient: _renew_subs 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f001d420 con 0x7fa7f81374e0 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7f8126680 con 0x7fa7f81374e0 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:39.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7f57fa700 4 set_mon_vals no callback set 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7eeffd700 10 monclient: dump: 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:39.290 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 10 monclient: _renew_subs 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.269+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7f8128780 con 0x7fa7f81374e0 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7fd2e6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7fd2e6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7fd2e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7f8131480 con 0x7fa7f81374e0 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa7f001d810 con 0x7fa7f81374e0 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7eeffd700 1 --2- 172.21.15.6:0/1131473519 >> v2:172.21.15.22:6812/142603 conn(0x7fa7d80357a0 0x7fa7d8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:39.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa7f0048ee0 con 0x7fa7f81374e0 2023-09-30T00:46:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7f6ffd700 1 --2- 172.21.15.6:0/1131473519 >> v2:172.21.15.22:6812/142603 conn(0x7fa7d80357a0 0x7fa7d8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.270+0000 7fa7f6ffd700 10 monclient: get_auth_request con 0x7fa7d80357a0 auth_method 0 2023-09-30T00:46:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.273+0000 7fa7f6ffd700 1 --2- 172.21.15.6:0/1131473519 >> v2:172.21.15.22:6812/142603 conn(0x7fa7d80357a0 0x7fa7d8037c50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8009db0 tx=0x7fa7e80091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.275+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa7f00076e0 con 0x7fa7f81374e0 2023-09-30T00:46:39.292 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.275+0000 7fa7eeffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:39.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.275+0000 7fa7eeffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.522+0000 7fa7fd2e6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.523+0000 7fa7fd2e6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.523+0000 7fa7fd2e6700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.523+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa7f8072b00 con 0x7fa7f81374e0 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.523+0000 7fa7eeffd700 1 -- 172.21.15.6:0/1131473519 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa7f0007320 con 0x7fa7f81374e0 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.523+0000 7fa7eeffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:39.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.523+0000 7fa7eeffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:39.525 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:39.525 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 >> v2:172.21.15.22:6812/142603 conn(0x7fa7d80357a0 msgr2=0x7fa7d8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:39.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/1131473519 >> v2:172.21.15.22:6812/142603 conn(0x7fa7d80357a0 0x7fa7d8037c50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa7e8009db0 tx=0x7fa7e80091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 10 monclient: shutdown 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 msgr2=0x7fa7f8129590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f8129590 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa7f000b6c0 tx=0x7fa7f0017000 comp rx=0 tx=0).stop 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 shutdown_connections 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/1131473519 >> v2:172.21.15.22:6812/142603 conn(0x7fa7d80357a0 0x7fa7d8037c50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa7f81374e0 0x7fa7f8129590 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa7f8098570 0x7fa7f812f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 --2- 172.21.15.6:0/1131473519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa7f81323a0 0x7fa7f8129050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:39.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 >> 172.21.15.6:0/1131473519 conn(0x7fa7f80a5480 msgr2=0x7fa7f80989f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 shutdown_connections 2023-09-30T00:46:39.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:39.525+0000 7fa7fd2e6700 1 -- 172.21.15.6:0/1131473519 wait complete. 2023-09-30T00:46:42.540 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:42.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.678+0000 7fabf1f86700 10 monclient: get_monmap_and_config 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.678+0000 7fabf1f86700 10 monclient: build_initial_monmap 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.678+0000 7fabf1f86700 10 monclient: monmap: 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:42.680080+0000 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:42.680080+0000 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:42.680 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:42.681 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:42.681 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:42.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.679+0000 7fabf1f86700 1 Processor -- start 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.679+0000 7fabf1f86700 1 -- start start 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.679+0000 7fabf1f86700 10 monclient: init 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient: _add_conns ranks=[1,3,2,0,4] 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fabec098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 0x7fabec137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): picked mon.noname-a con 0x7fabec132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): picked mon.noname-b con 0x7fabec137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabec0a1000 con 0x7fabec098670 2023-09-30T00:46:42.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabec099150 con 0x7fabec132d00 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0a0a50 con 0x7fabec137b60 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabf1f86700 10 monclient(hunting): authenticate will time out at 1398.978276s 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 0x7fabec137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 0x7fabec137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42194/0 (socket says 172.21.15.6:42194) 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 1 -- 172.21.15.6:0/1266637235 learned_addr learned my addr 172.21.15.6:0/1266637235 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): get_auth_request con 0x7fabec137b60 auth_method 0 2023-09-30T00:46:42.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): get_auth_request con 0x7fabec132d00 auth_method 0 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:42.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient(hunting): handle_auth_done global_id 4499 payload 274 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 1 -- 172.21.15.6:0/1266637235 >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 msgr2=0x7fabec137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 1 --2- 172.21.15.6:0/1266637235 >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 0x7fabec137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 1 -- 172.21.15.6:0/1266637235 >> v1:172.21.15.22:6789/0 conn(0x7fabec098670 legacy=0x7fabec098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 1 monclient: found mon.noname-b 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabebfff700 1 -- 172.21.15.6:0/1266637235 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabec0a0730 con 0x7fabec137b60 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.680+0000 7fabeaffd700 1 --2- 172.21.15.6:0/1266637235 >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 0x7fabec137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabebfff700 1 -- 172.21.15.6:0/1266637235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 msgr2=0x7fabec139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabebfff700 1 --2- 172.21.15.6:0/1266637235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec139f40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009e60 tx=0x7fabe4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae589c69bdb371 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:42.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 1 -- 172.21.15.6:0/1266637235 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400e070 con 0x7fabec137b60 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: dump: 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:42.689 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: _finish_auth 0 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 1 -- 172.21.15.6:0/1266637235 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe400bd30 con 0x7fabec137b60 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 5 monclient: authenticate success, global_id 4499 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 1 -- 172.21.15.6:0/1266637235 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe400bea0 con 0x7fabec137b60 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 4 set_mon_vals no callback set 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabea7fc700 10 monclient: dump: 2023-09-30T00:46:42.690 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 10 monclient: shutdown 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:42.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 -- 172.21.15.6:0/1266637235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 msgr2=0x7fabec139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1266637235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec139f40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009e60 tx=0x7fabe4009780 comp rx=0 tx=0).stop 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 -- 172.21.15.6:0/1266637235 shutdown_connections 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1266637235 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec139f40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1266637235 >> v2:172.21.15.6:3300/0 conn(0x7fabec132d00 0x7fabec137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 -- 172.21.15.6:0/1266637235 >> 172.21.15.6:0/1266637235 conn(0x7fabec0a5480 msgr2=0x7fabec0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 -- 172.21.15.6:0/1266637235 shutdown_connections 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.683+0000 7fabf1f86700 1 -- 172.21.15.6:0/1266637235 wait complete. 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.684+0000 7fabf1f86700 10 monclient: build_initial_monmap 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.684+0000 7fabf1f86700 10 monclient: monmap: 2023-09-30T00:46:42.692 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:42.693 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:42.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.684+0000 7fabf1f86700 1 Processor -- start 2023-09-30T00:46:42.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 -- start start 2023-09-30T00:46:42.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient: init 2023-09-30T00:46:42.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec098670 0x7fabec09b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): picked mon.noname-b con 0x7fabec098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 0x7fabec09cc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): picked mon.noname-a con 0x7fabec132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec136a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): picked mon.noname-c con 0x7fabec137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec0a3380 con 0x7fabec132d00 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabec09e500 con 0x7fabec098670 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabec09f2a0 con 0x7fabec137b60 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabf1f86700 10 monclient(hunting): authenticate will time out at 1398.983496s 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec136a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 0x7fabec09cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec136a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:47458/0 (socket says 172.21.15.6:47458) 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 0x7fabec09cc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42210/0 (socket says 172.21.15.6:42210) 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 -- 172.21.15.6:0/1026339368 learned_addr learned my addr 172.21.15.6:0/1026339368 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): get_auth_request con 0x7fabec137b60 auth_method 0 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.699 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): get_auth_request con 0x7fabec132d00 auth_method 0 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec098670 0x7fabec09b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:42.700 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): get_auth_request con 0x7fabec098670 auth_method 0 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient(hunting): handle_auth_done global_id 4502 payload 824 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 -- 172.21.15.6:0/1026339368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec098670 msgr2=0x7fabec09b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec098670 0x7fabec09b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 -- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 msgr2=0x7fabec09cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 0x7fabec09cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 monclient: found mon.noname-c 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabebfff700 1 -- 172.21.15.6:0/1026339368 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabe00094f0 con 0x7fabec137b60 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.685+0000 7fabeaffd700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 0x7fabec09cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabebfff700 1 -- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 msgr2=0x7fabec136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabebfff700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec136a60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fabe400b6c0 tx=0x7fabe4018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4007000 con 0x7fabec137b60 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: dump: 2023-09-30T00:46:42.702 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:42.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabe4021070 con 0x7fabec137b60 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe4026430 con 0x7fabec137b60 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabd37fe700 10 monclient: dump: 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:42.704 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 5 monclient: authenticate success, global_id 4502 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 10 monclient: _renew_subs 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabe4009a20 con 0x7fabec137b60 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabe9ffb700 4 set_mon_vals no callback set 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 10 monclient: _renew_subs 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.688+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabec09d900 con 0x7fabec137b60 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fabe4023030 con 0x7fabec137b60 2023-09-30T00:46:42.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabd37fe700 1 --2- 172.21.15.6:0/1026339368 >> v2:172.21.15.22:6812/142603 conn(0x7fabd80357a0 0x7fabd8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fabe4049a20 con 0x7fabec137b60 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabf1f86700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabf1f86700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabf1f86700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabec072b00 con 0x7fabec137b60 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabeb7fe700 1 --2- 172.21.15.6:0/1026339368 >> v2:172.21.15.22:6812/142603 conn(0x7fabd80357a0 0x7fabd8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.689+0000 7fabeb7fe700 10 monclient: get_auth_request con 0x7fabd80357a0 auth_method 0 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.692+0000 7fabeb7fe700 1 --2- 172.21.15.6:0/1026339368 >> v2:172.21.15.22:6812/142603 conn(0x7fabd80357a0 0x7fabd8037c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0039a0 tx=0x7fabdc007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.694+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fabe401c420 con 0x7fabec137b60 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.694+0000 7fabd37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:42.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.694+0000 7fabd37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:42.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.938+0000 7fabf1f86700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:42.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.938+0000 7fabf1f86700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:42.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.938+0000 7fabf1f86700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:42.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.938+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fabec063920 con 0x7fabec137b60 2023-09-30T00:46:42.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.939+0000 7fabd37fe700 1 -- 172.21.15.6:0/1026339368 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fabe401b3a0 con 0x7fabec137b60 2023-09-30T00:46:42.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.939+0000 7fabd37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:42.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.939+0000 7fabd37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:42.941 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:42.941 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:42.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 >> v2:172.21.15.22:6812/142603 conn(0x7fabd80357a0 msgr2=0x7fabd8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:42.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1026339368 >> v2:172.21.15.22:6812/142603 conn(0x7fabd80357a0 0x7fabd8037c50 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fabdc0039a0 tx=0x7fabdc007ea0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 10 monclient: shutdown 2023-09-30T00:46:42.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:42.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 msgr2=0x7fabec136a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec136a60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fabe400b6c0 tx=0x7fabe4018000 comp rx=0 tx=0).stop 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 shutdown_connections 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1026339368 >> v2:172.21.15.22:6812/142603 conn(0x7fabd80357a0 0x7fabd8037c50 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fabec137b60 0x7fabec136a60 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fabec098670 0x7fabec09b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 --2- 172.21.15.6:0/1026339368 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fabec132d00 0x7fabec09cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 >> 172.21.15.6:0/1026339368 conn(0x7fabec0a5480 msgr2=0x7fabec097620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 shutdown_connections 2023-09-30T00:46:42.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:42.941+0000 7fabf1f86700 1 -- 172.21.15.6:0/1026339368 wait complete. 2023-09-30T00:46:45.956 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:46.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.093+0000 7f0427029700 10 monclient: get_monmap_and_config 2023-09-30T00:46:46.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.093+0000 7f0427029700 10 monclient: build_initial_monmap 2023-09-30T00:46:46.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.093+0000 7f0427029700 10 monclient: monmap: 2023-09-30T00:46:46.094 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:46.094 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:46.094933+0000 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:46.094933+0000 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:46.095 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.094+0000 7f0427029700 1 Processor -- start 2023-09-30T00:46:46.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.094+0000 7f0427029700 1 -- start start 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.094+0000 7f0427029700 10 monclient: init 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient: _add_conns ranks=[2,4,3,0,1] 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 0x7f0420132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): picked mon.noname-b con 0x7f0420098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f0420098a40 0x7f042012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): picked mon.noname-c con 0x7f0420098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0420098e90 0x7f042012f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): picked mon.noname-a con 0x7f0420098e90 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:46.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04200a0a50 con 0x7f0420098e90 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04200a1000 con 0x7f0420098a40 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04200a0730 con 0x7f0420098670 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0427029700 10 monclient(hunting): authenticate will time out at 1402.393240s 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 0x7f0420132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0420098e90 0x7f042012f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 0x7f0420132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:47464/0 (socket says 172.21.15.6:47464) 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 -- 172.21.15.6:0/1177240193 learned_addr learned my addr 172.21.15.6:0/1177240193 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): get_auth_request con 0x7f0420098670 auth_method 0 2023-09-30T00:46:46.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): get_auth_request con 0x7f0420098e90 auth_method 0 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:46.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f041ffff700 1 --2- 172.21.15.6:0/1177240193 >> v2:172.21.15.22:3300/0 conn(0x7f0420098a40 0x7f042012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f041ffff700 10 monclient(hunting): get_auth_request con 0x7f0420098a40 auth_method 0 2023-09-30T00:46:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f041ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f041ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f041ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient(hunting): handle_auth_done global_id 4505 payload 274 2023-09-30T00:46:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 -- 172.21.15.6:0/1177240193 >> v2:172.21.15.22:3300/0 conn(0x7f0420098a40 msgr2=0x7f042012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 --2- 172.21.15.6:0/1177240193 >> v2:172.21.15.22:3300/0 conn(0x7f0420098a40 0x7f042012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 -- 172.21.15.6:0/1177240193 >> v2:172.21.15.6:3300/0 conn(0x7f0420098e90 msgr2=0x7f042012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:46.104 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 --2- 172.21.15.6:0/1177240193 >> v2:172.21.15.6:3300/0 conn(0x7f0420098e90 0x7f042012f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient: _finish_hunting 0 2023-09-30T00:46:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 monclient: found mon.noname-b 2023-09-30T00:46:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f0424dc5700 1 -- 172.21.15.6:0/1177240193 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0420068df0 con 0x7f0420098670 2023-09-30T00:46:46.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.095+0000 7f04255c6700 1 --2- 172.21.15.6:0/1177240193 >> v2:172.21.15.6:3300/0 conn(0x7f0420098e90 0x7f042012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.097+0000 7f0424dc5700 1 -- 172.21.15.6:0/1177240193 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 msgr2=0x7f0420132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.097+0000 7f0424dc5700 1 --2- 172.21.15.6:0/1177240193 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 0x7f0420132d00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f042006a400 tx=0x7f0410009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=db8633a8213d99e0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 1 -- 172.21.15.6:0/1177240193 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0410009e70 con 0x7f0420098670 2023-09-30T00:46:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:46.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: dump: 2023-09-30T00:46:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:46.107 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:46.108 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:46.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 1 -- 172.21.15.6:0/1177240193 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f041000ba70 con 0x7f0420098670 2023-09-30T00:46:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 5 monclient: authenticate success, global_id 4505 2023-09-30T00:46:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:46.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:46.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 4 set_mon_vals no callback set 2023-09-30T00:46:46.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 1 -- 172.21.15.6:0/1177240193 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f041000be30 con 0x7f0420098670 2023-09-30T00:46:46.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:46.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:46.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f041f7fe700 10 monclient: dump: 2023-09-30T00:46:46.110 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:46.111 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:46.113 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:46.113 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 10 monclient: shutdown 2023-09-30T00:46:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 -- 172.21.15.6:0/1177240193 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 msgr2=0x7f0420132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 --2- 172.21.15.6:0/1177240193 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 0x7f0420132d00 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f042006a400 tx=0x7f0410009250 comp rx=0 tx=0).stop 2023-09-30T00:46:46.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 -- 172.21.15.6:0/1177240193 shutdown_connections 2023-09-30T00:46:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 --2- 172.21.15.6:0/1177240193 >> v2:172.21.15.6:3300/0 conn(0x7f0420098e90 0x7f042012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 --2- 172.21.15.6:0/1177240193 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098670 0x7f0420132d00 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 --2- 172.21.15.6:0/1177240193 >> v2:172.21.15.22:3300/0 conn(0x7f0420098a40 0x7f042012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 -- 172.21.15.6:0/1177240193 >> 172.21.15.6:0/1177240193 conn(0x7f04200a5480 msgr2=0x7f04200a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 -- 172.21.15.6:0/1177240193 shutdown_connections 2023-09-30T00:46:46.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.098+0000 7f0427029700 1 -- 172.21.15.6:0/1177240193 wait complete. 2023-09-30T00:46:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.099+0000 7f0427029700 10 monclient: build_initial_monmap 2023-09-30T00:46:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.099+0000 7f0427029700 10 monclient: monmap: 2023-09-30T00:46:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:46.116 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:46.117 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:46.118 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:46.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.099+0000 7f0427029700 1 Processor -- start 2023-09-30T00:46:46.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 -- start start 2023-09-30T00:46:46.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient: init 2023-09-30T00:46:46.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:46.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:46:46.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0420098670 0x7f042009ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): picked mon.noname-b con 0x7f0420098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 0x7f04201417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): picked mon.noname-c con 0x7f0420098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0420098e90 0x7f0420141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): picked mon.noname-a con 0x7f0420098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:46.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04200a1500 con 0x7f0420098e90 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f042009df50 con 0x7f0420098670 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f042009e230 con 0x7f0420098a40 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0427029700 10 monclient(hunting): authenticate will time out at 1402.398648s 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 0x7f04201417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 0x7f04201417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:47470/0 (socket says 172.21.15.6:47470) 2023-09-30T00:46:46.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 1 -- 172.21.15.6:0/3815337301 learned_addr learned my addr 172.21.15.6:0/3815337301 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): get_auth_request con 0x7f0420098a40 auth_method 0 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0420098e90 0x7f0420141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): get_auth_request con 0x7f0420098e90 auth_method 0 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0424dc5700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0420098670 0x7f042009ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f041ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f04255c6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:46.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0424dc5700 10 monclient(hunting): get_auth_request con 0x7f0420098670 auth_method 0 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0424dc5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0424dc5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.100+0000 7f0424dc5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f0424dc5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f0424dc5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f0424dc5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 10 monclient(hunting): handle_auth_done global_id 4508 payload 824 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 1 -- 172.21.15.6:0/3815337301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0420098670 msgr2=0x7f042009ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:46.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0420098670 0x7f042009ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 1 -- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0420098e90 msgr2=0x7f0420141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0420098e90 0x7f0420141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 10 monclient: _finish_hunting 0 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 1 monclient: found mon.noname-c 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f041ffff700 1 -- 172.21.15.6:0/3815337301 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0418009a20 con 0x7f0420098a40 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.101+0000 7f04255c6700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0420098e90 0x7f0420141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:46.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041ffff700 1 -- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 msgr2=0x7f04201417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041ffff700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 0x7f04201417d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f040800d890 tx=0x7f0408008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408012800 con 0x7f0420098a40 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: dump: 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:46.134 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: _finish_auth 0 2023-09-30T00:46:46.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 5 monclient: authenticate success, global_id 4508 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0408012960 con 0x7f0420098a40 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 10 monclient: _renew_subs 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0408012ce0 con 0x7f0420098a40 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04100094f0 con 0x7f0420098a40 2023-09-30T00:46:46.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041effd700 4 set_mon_vals no callback set 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f041cff9700 10 monclient: dump: 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:46.137 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 10 monclient: _renew_subs 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.103+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f042012fa80 con 0x7f0420098a40 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0408014640 con 0x7f0420098a40 2023-09-30T00:46:46.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f0427029700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f0427029700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f0427029700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f041cff9700 1 --2- 172.21.15.6:0/3815337301 >> v2:172.21.15.22:6812/142603 conn(0x7f04000357a0 0x7f0400037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0420098110 con 0x7f0420098a40 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0408047fa0 con 0x7f0420098a40 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.104+0000 7f0424dc5700 1 --2- 172.21.15.6:0/3815337301 >> v2:172.21.15.22:6812/142603 conn(0x7f04000357a0 0x7f0400037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:46.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.105+0000 7f0424dc5700 10 monclient: get_auth_request con 0x7f04000357a0 auth_method 0 2023-09-30T00:46:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.107+0000 7f0424dc5700 1 --2- 172.21.15.6:0/3815337301 >> v2:172.21.15.22:6812/142603 conn(0x7f04000357a0 0x7f0400037c50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f042006a400 tx=0x7f0410017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.109+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f040801c950 con 0x7f0420098a40 2023-09-30T00:46:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.109+0000 7f041cff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:46.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.109+0000 7f041cff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:46.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f0427029700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f0427029700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f0427029700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f042009d060 con 0x7f0420098a40 2023-09-30T00:46:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f041cff9700 1 -- 172.21.15.6:0/3815337301 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0408001b40 con 0x7f0420098a40 2023-09-30T00:46:46.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f041cff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:46.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.359+0000 7f041cff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:46.361 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:46.361 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:46.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 >> v2:172.21.15.22:6812/142603 conn(0x7f04000357a0 msgr2=0x7f0400037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:46.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 --2- 172.21.15.6:0/3815337301 >> v2:172.21.15.22:6812/142603 conn(0x7f04000357a0 0x7f0400037c50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f042006a400 tx=0x7f0410017000 comp rx=0 tx=0).stop 2023-09-30T00:46:46.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 10 monclient: shutdown 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 msgr2=0x7f04201417d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 0x7f04201417d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f040800d890 tx=0x7f0408008040 comp rx=0 tx=0).stop 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 shutdown_connections 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 --2- 172.21.15.6:0/3815337301 >> v2:172.21.15.22:6812/142603 conn(0x7f04000357a0 0x7f0400037c50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0420098e90 0x7f0420141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0420098670 0x7f042009ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 --2- 172.21.15.6:0/3815337301 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0420098a40 0x7f04201417d0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:46.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 >> 172.21.15.6:0/3815337301 conn(0x7f04200a5480 msgr2=0x7f0420133b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:46.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 shutdown_connections 2023-09-30T00:46:46.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:46.361+0000 7f0427029700 1 -- 172.21.15.6:0/3815337301 wait complete. 2023-09-30T00:46:49.376 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:49.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.514+0000 7f4d9d703700 10 monclient: get_monmap_and_config 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.514+0000 7f4d9d703700 10 monclient: build_initial_monmap 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.514+0000 7f4d9d703700 10 monclient: monmap: 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:49.515458+0000 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:49.515458+0000 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:49.515 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:49.516 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:49.516 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.514+0000 7f4d9d703700 1 Processor -- start 2023-09-30T00:46:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.514+0000 7f4d9d703700 1 -- start start 2023-09-30T00:46:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.514+0000 7f4d9d703700 10 monclient: init 2023-09-30T00:46:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T00:46:49.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4d981323a0 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 0x7f4d980989f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient(hunting): picked mon.noname-b con 0x7f4d98098570 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4d981374e0 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d98096a80 con 0x7f4d981323a0 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d98096d60 con 0x7f4d981374e0 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d98132e80 con 0x7f4d98098570 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:49.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d9d703700 10 monclient(hunting): authenticate will time out at 1405.813519s 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d96ffd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4d981323a0 0x7f4d98132770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41688/0 (socket says 172.21.15.6:41688) 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d96ffd700 1 -- 172.21.15.6:0/2377712090 learned_addr learned my addr 172.21.15.6:0/2377712090 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 1 --2- 172.21.15.6:0/2377712090 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 0x7f4d980989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): get_auth_request con 0x7f4d98098570 auth_method 0 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient(hunting): handle_auth_done global_id 4511 payload 274 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 1 -- 172.21.15.6:0/2377712090 >> v1:172.21.15.22:6789/0 conn(0x7f4d981374e0 legacy=0x7f4d981398c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 1 -- 172.21.15.6:0/2377712090 >> v1:172.21.15.6:6789/0 conn(0x7f4d981323a0 legacy=0x7f4d98132770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient: _finish_hunting 0 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 1 monclient: found mon.noname-b 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:49.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.515+0000 7f4d967fc700 1 -- 172.21.15.6:0/2377712090 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d980a1c30 con 0x7f4d98098570 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d967fc700 1 -- 172.21.15.6:0/2377712090 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 msgr2=0x7f4d980989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d967fc700 1 --2- 172.21.15.6:0/2377712090 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 0x7f4d980989f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009940 tx=0x7f4d800091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12538c9f0d1c5769 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 1 -- 172.21.15.6:0/2377712090 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8000b9f0 con 0x7f4d98098570 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: dump: 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.527 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 1 -- 172.21.15.6:0/2377712090 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d8000bb50 con 0x7f4d98098570 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 5 monclient: authenticate success, global_id 4511 2023-09-30T00:46:49.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 1 -- 172.21.15.6:0/2377712090 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d80011420 con 0x7f4d98098570 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 4 set_mon_vals no callback set 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d95ffb700 10 monclient: dump: 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:49.529 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 10 monclient: shutdown 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:49.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 1 -- 172.21.15.6:0/2377712090 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 msgr2=0x7f4d980989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.518+0000 7f4d9d703700 1 --2- 172.21.15.6:0/2377712090 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 0x7f4d980989f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4d80009940 tx=0x7f4d800091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 1 -- 172.21.15.6:0/2377712090 shutdown_connections 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 1 --2- 172.21.15.6:0/2377712090 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d98098570 0x7f4d980989f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 1 -- 172.21.15.6:0/2377712090 >> 172.21.15.6:0/2377712090 conn(0x7f4d980a5480 msgr2=0x7f4d980a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 1 -- 172.21.15.6:0/2377712090 shutdown_connections 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 1 -- 172.21.15.6:0/2377712090 wait complete. 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 10 monclient: build_initial_monmap 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.519+0000 7f4d9d703700 10 monclient: monmap: 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:49.531 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 Processor -- start 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 -- start start 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient: init 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:49.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 0x7f4d9809ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): picked mon.noname-a con 0x7f4d98098570 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 0x7f4d98099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): picked mon.noname-c con 0x7f4d981323a0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d981374e0 0x7f4d980997d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): picked mon.noname-b con 0x7f4d981374e0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d980a11b0 con 0x7f4d98098570 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9813adb0 con 0x7f4d981374e0 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:49.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d9809c0c0 con 0x7f4d981323a0 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d9d703700 10 monclient(hunting): authenticate will time out at 1405.818670s 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 0x7f4d9809ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 0x7f4d98099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 0x7f4d9809ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46872/0 (socket says 172.21.15.6:46872) 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 1 -- 172.21.15.6:0/3306732478 learned_addr learned my addr 172.21.15.6:0/3306732478 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): get_auth_request con 0x7f4d98098570 auth_method 0 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 0x7f4d98099290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:59300/0 (socket says 172.21.15.6:59300) 2023-09-30T00:46:49.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): get_auth_request con 0x7f4d981323a0 auth_method 0 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d977fe700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d981374e0 0x7f4d980997d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d96ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d967fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d977fe700 10 monclient(hunting): get_auth_request con 0x7f4d981374e0 auth_method 0 2023-09-30T00:46:49.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d977fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d977fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.520+0000 7f4d977fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d977fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d977fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d977fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 10 monclient(hunting): handle_auth_done global_id 4514 payload 824 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 1 -- 172.21.15.6:0/3306732478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d981374e0 msgr2=0x7f4d980997d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d981374e0 0x7f4d980997d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 1 -- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 msgr2=0x7f4d9809ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 0x7f4d9809ed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 10 monclient: _finish_hunting 0 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 1 monclient: found mon.noname-c 2023-09-30T00:46:49.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d967fc700 1 -- 172.21.15.6:0/3306732478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d80009420 con 0x7f4d981323a0 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.521+0000 7f4d96ffd700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 0x7f4d9809ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d967fc700 1 -- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 msgr2=0x7f4d98099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d967fc700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 0x7f4d98099290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d980a53e0 tx=0x7f4d80016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8000bed0 con 0x7f4d981323a0 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: dump: 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:49.537 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: _finish_auth 0 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d80011e50 con 0x7f4d981323a0 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:49.538 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d8001a8d0 con 0x7f4d981323a0 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 5 monclient: authenticate success, global_id 4514 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 10 monclient: _renew_subs 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d98136b40 con 0x7f4d981323a0 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d957fa700 4 set_mon_vals no callback set 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d8effd700 10 monclient: dump: 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:49.539 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 10 monclient: _renew_subs 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:49.540 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.523+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d9809a500 con 0x7f4d981323a0 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4d8001ac40 con 0x7f4d981323a0 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d8effd700 1 --2- 172.21.15.6:0/3306732478 >> v2:172.21.15.22:6812/142603 conn(0x7f4d780357a0 0x7f4d78037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d9d703700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d9d703700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d9d703700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4d80049a90 con 0x7f4d981323a0 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.524+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d98133120 con 0x7f4d981323a0 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.525+0000 7f4d96ffd700 1 --2- 172.21.15.6:0/3306732478 >> v2:172.21.15.22:6812/142603 conn(0x7f4d780357a0 0x7f4d78037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:49.541 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.525+0000 7f4d96ffd700 10 monclient: get_auth_request con 0x7f4d780357a0 auth_method 0 2023-09-30T00:46:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.527+0000 7f4d96ffd700 1 --2- 172.21.15.6:0/3306732478 >> v2:172.21.15.22:6812/142603 conn(0x7f4d780357a0 0x7f4d78037c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d880052e0 tx=0x7f4d88004a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.529+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4d98133120 con 0x7f4d981323a0 2023-09-30T00:46:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.529+0000 7f4d8effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:49.542 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.530+0000 7f4d8effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.776+0000 7f4d9d703700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.776+0000 7f4d9d703700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.776+0000 7f4d9d703700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:49.777 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.776+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d98133300 con 0x7f4d981323a0 2023-09-30T00:46:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.777+0000 7f4d8effd700 1 -- 172.21.15.6:0/3306732478 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4d98133300 con 0x7f4d981323a0 2023-09-30T00:46:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.777+0000 7f4d8effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:49.778 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.777+0000 7f4d8effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:49.778 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:49.778 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 >> v2:172.21.15.22:6812/142603 conn(0x7f4d780357a0 msgr2=0x7f4d78037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 --2- 172.21.15.6:0/3306732478 >> v2:172.21.15.22:6812/142603 conn(0x7f4d780357a0 0x7f4d78037c50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4d880052e0 tx=0x7f4d88004a80 comp rx=0 tx=0).stop 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 10 monclient: shutdown 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 msgr2=0x7f4d98099290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 0x7f4d98099290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f4d980a53e0 tx=0x7f4d80016000 comp rx=0 tx=0).stop 2023-09-30T00:46:49.780 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 shutdown_connections 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 --2- 172.21.15.6:0/3306732478 >> v2:172.21.15.22:6812/142603 conn(0x7f4d780357a0 0x7f4d78037c50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4d981374e0 0x7f4d980997d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4d98098570 0x7f4d9809ed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 --2- 172.21.15.6:0/3306732478 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4d981323a0 0x7f4d98099290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 >> 172.21.15.6:0/3306732478 conn(0x7f4d980a5480 msgr2=0x7f4d98131930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 shutdown_connections 2023-09-30T00:46:49.781 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:49.779+0000 7f4d9d703700 1 -- 172.21.15.6:0/3306732478 wait complete. 2023-09-30T00:46:52.794 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:52.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.930+0000 7ff8880f2700 10 monclient: get_monmap_and_config 2023-09-30T00:46:52.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.930+0000 7ff8880f2700 10 monclient: build_initial_monmap 2023-09-30T00:46:52.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.930+0000 7ff8880f2700 10 monclient: monmap: 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:52.932047+0000 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:52.932047+0000 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:52.932 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:52.933 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:52.933 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.931+0000 7ff8880f2700 1 Processor -- start 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.931+0000 7ff8880f2700 1 -- start start 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.931+0000 7ff8880f2700 10 monclient: init 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient: _add_conns ranks=[0,2,3,1,4] 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff880098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880132d00 0x7ff880137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.937 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): picked mon.noname-b con 0x7ff880132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff880137b60 0x7ff880139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): picked mon.noname-a con 0x7ff880137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8800a1000 con 0x7ff880098670 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff880099150 con 0x7ff880137b60 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800a0a50 con 0x7ff880132d00 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff8880f2700 10 monclient(hunting): authenticate will time out at 1409.230350s 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff885e8e700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff880098670 0x7ff880098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41692/0 (socket says 172.21.15.6:41692) 2023-09-30T00:46:52.938 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff880137b60 0x7ff880139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff885e8e700 1 -- 172.21.15.6:0/3187762811 learned_addr learned my addr 172.21.15.6:0/3187762811 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880132d00 0x7ff880137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): get_auth_request con 0x7ff880132d00 auth_method 0 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): get_auth_request con 0x7ff880137b60 auth_method 0 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.939 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient(hunting): handle_auth_done global_id 4515 payload 274 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 1 -- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880132d00 msgr2=0x7ff880137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880132d00 0x7ff880137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 1 -- 172.21.15.6:0/3187762811 >> v1:172.21.15.6:6789/0 conn(0x7ff880098670 legacy=0x7ff880098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient: _finish_hunting 0 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 1 monclient: found mon.noname-a 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88668f700 1 -- 172.21.15.6:0/3187762811 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8800a0730 con 0x7ff880137b60 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.932+0000 7ff88568d700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880132d00 0x7ff880137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff88668f700 1 -- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880137b60 msgr2=0x7ff880139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff88668f700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880137b60 0x7ff880139f40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff878009e90 tx=0x7ff878009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e830ac2608397 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 1 -- 172.21.15.6:0/3187762811 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87800e070 con 0x7ff880137b60 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: dump: 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:52.942 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: _finish_auth 0 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 5 monclient: authenticate success, global_id 4515 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 1 -- 172.21.15.6:0/3187762811 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87800bdc0 con 0x7ff880137b60 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 4 set_mon_vals no callback set 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 1 -- 172.21.15.6:0/3187762811 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff878012660 con 0x7ff880137b60 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:52.943 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff884e8c700 10 monclient: dump: 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 10 monclient: shutdown 2023-09-30T00:46:52.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 -- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880137b60 msgr2=0x7ff880139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880137b60 0x7ff880139f40 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7ff878009e90 tx=0x7ff878009780 comp rx=0 tx=0).stop 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 -- 172.21.15.6:0/3187762811 shutdown_connections 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880137b60 0x7ff880139f40 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3187762811 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880132d00 0x7ff880137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 -- 172.21.15.6:0/3187762811 >> 172.21.15.6:0/3187762811 conn(0x7ff8800a5480 msgr2=0x7ff8800a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.935+0000 7ff8880f2700 1 -- 172.21.15.6:0/3187762811 shutdown_connections 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.936+0000 7ff8880f2700 1 -- 172.21.15.6:0/3187762811 wait complete. 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.936+0000 7ff8880f2700 10 monclient: build_initial_monmap 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.936+0000 7ff8880f2700 10 monclient: monmap: 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:52.945 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.936+0000 7ff8880f2700 1 Processor -- start 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.936+0000 7ff8880f2700 1 -- start start 2023-09-30T00:46:52.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.936+0000 7ff8880f2700 10 monclient: init 2023-09-30T00:46:52.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:52.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:52.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 0x7ff880137500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): picked mon.noname-a con 0x7ff880098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff880132d00 0x7ff88009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): picked mon.noname-b con 0x7ff880132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880137b60 0x7ff88009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): picked mon.noname-c con 0x7ff880137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff880133e60 con 0x7ff880098670 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800a1500 con 0x7ff880132d00 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:52.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8800a3380 con 0x7ff880137b60 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff8880f2700 10 monclient(hunting): authenticate will time out at 1409.235367s 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 0x7ff880137500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 0x7ff880137500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46896/0 (socket says 172.21.15.6:46896) 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880137b60 0x7ff88009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 -- 172.21.15.6:0/3383019623 learned_addr learned my addr 172.21.15.6:0/3383019623 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): get_auth_request con 0x7ff880098670 auth_method 0 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): get_auth_request con 0x7ff880137b60 auth_method 0 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:52.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88568d700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff880132d00 0x7ff88009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient(hunting): handle_auth_done global_id 4518 payload 824 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 -- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880137b60 msgr2=0x7ff88009cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880137b60 0x7ff88009cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 -- 172.21.15.6:0/3383019623 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff880132d00 msgr2=0x7ff88009c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff880132d00 0x7ff88009c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient: _finish_hunting 0 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 monclient: found mon.noname-a 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:52.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff885e8e700 1 -- 172.21.15.6:0/3383019623 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff874009420 con 0x7ff880098670 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88568d700 10 monclient: get_auth_request con 0x7ff880132d00 auth_method 0 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88568d700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff880132d00 0x7ff88009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.937+0000 7ff88668f700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880137b60 0x7ff88009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff885e8e700 1 -- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 msgr2=0x7ff880137500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff885e8e700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 0x7ff880137500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c610 tx=0x7ff87c004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c005d40 con 0x7ff880098670 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: dump: 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:52.955 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff87c0070d0 con 0x7ff880098670 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 5 monclient: authenticate success, global_id 4518 2023-09-30T00:46:52.956 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 10 monclient: _renew_subs 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff878009a20 con 0x7ff880098670 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff87c007490 con 0x7ff880098670 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff871ffb700 10 monclient: dump: 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:52.957 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff873fff700 4 set_mon_vals no callback set 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 10 monclient: _renew_subs 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.940+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8801341f0 con 0x7ff880098670 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff87c01c440 con 0x7ff880098670 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff8880f2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff8880f2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff8880f2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff880072b00 con 0x7ff880098670 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff871ffb700 1 --2- 172.21.15.6:0/3383019623 >> v2:172.21.15.22:6812/142603 conn(0x7ff8640357a0 0x7ff864037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.941+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff87c046ea0 con 0x7ff880098670 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.942+0000 7ff88568d700 1 --2- 172.21.15.6:0/3383019623 >> v2:172.21.15.22:6812/142603 conn(0x7ff8640357a0 0x7ff864037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.942+0000 7ff88568d700 10 monclient: get_auth_request con 0x7ff8640357a0 auth_method 0 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.944+0000 7ff88568d700 1 --2- 172.21.15.6:0/3383019623 >> v2:172.21.15.22:6812/142603 conn(0x7ff8640357a0 0x7ff864037c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff8740093f0 tx=0x7ff874008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.946+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff87c00e680 con 0x7ff880098670 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.946+0000 7ff871ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:52.959 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:52.946+0000 7ff871ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff8880f2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff8880f2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff8880f2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff880133c10 con 0x7ff880098670 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff871ffb700 1 -- 172.21.15.6:0/3383019623 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff87c011360 con 0x7ff880098670 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff871ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:53.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.199+0000 7ff871ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:53.201 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:53.201 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:53.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.201+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 >> v2:172.21.15.22:6812/142603 conn(0x7ff8640357a0 msgr2=0x7ff864037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.201+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3383019623 >> v2:172.21.15.22:6812/142603 conn(0x7ff8640357a0 0x7ff864037c50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff8740093f0 tx=0x7ff874008040 comp rx=0 tx=0).stop 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.201+0000 7ff8880f2700 10 monclient: shutdown 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.201+0000 7ff8880f2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.201+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 msgr2=0x7ff880137500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.201+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 0x7ff880137500 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff87c00c610 tx=0x7ff87c004a80 comp rx=0 tx=0).stop 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 shutdown_connections 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3383019623 >> v2:172.21.15.22:6812/142603 conn(0x7ff8640357a0 0x7ff864037c50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff880137b60 0x7ff88009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff880098670 0x7ff880137500 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 --2- 172.21.15.6:0/3383019623 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff880132d00 0x7ff88009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:53.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 >> 172.21.15.6:0/3383019623 conn(0x7ff8800a5480 msgr2=0x7ff880097620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:53.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 shutdown_connections 2023-09-30T00:46:53.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:53.202+0000 7ff8880f2700 1 -- 172.21.15.6:0/3383019623 wait complete. 2023-09-30T00:46:56.216 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.352+0000 7fe6ceb31700 10 monclient: get_monmap_and_config 2023-09-30T00:46:56.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.352+0000 7fe6ceb31700 10 monclient: build_initial_monmap 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.352+0000 7fe6ceb31700 10 monclient: monmap: 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:56.354024+0000 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:56.354024+0000 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:56.354 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:56.355 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:56.355 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:56.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.353+0000 7fe6ceb31700 1 Processor -- start 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.353+0000 7fe6ceb31700 1 -- start start 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.353+0000 7fe6ceb31700 10 monclient: init 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 0x7fe6c8132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient(hunting): picked mon.noname-b con 0x7fe6c8098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fe6c8098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fe6c8098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c80a1000 con 0x7fe6c8098e90 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6c812fc00 con 0x7fe6c8098a40 2023-09-30T00:46:56.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c80a0a50 con 0x7fe6c8098670 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6ceb31700 10 monclient(hunting): authenticate will time out at 1412.652264s 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cd0ce700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fe6c8098e90 0x7fe6c812f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41696/0 (socket says 172.21.15.6:41696) 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 0x7fe6c8132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cd0ce700 1 -- 172.21.15.6:0/55178601 learned_addr learned my addr 172.21.15.6:0/55178601 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): get_auth_request con 0x7fe6c8098670 auth_method 0 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:56.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_done global_id 4523 payload 274 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/55178601 >> v1:172.21.15.22:6789/0 conn(0x7fe6c8098a40 legacy=0x7fe6c812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/55178601 >> v1:172.21.15.6:6789/0 conn(0x7fe6c8098e90 legacy=0x7fe6c812f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient: _finish_hunting 0 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 1 monclient: found mon.noname-b 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:56.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.354+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/55178601 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6c80a0730 con 0x7fe6c8098670 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.356+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/55178601 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 msgr2=0x7fe6c8132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.356+0000 7fe6cc8cd700 1 --2- 172.21.15.6:0/55178601 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 0x7fe6c8132d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009910 tx=0x7fe6b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=262783983fc37c5f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 1 -- 172.21.15.6:0/55178601 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800b9f0 con 0x7fe6c8098670 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: dump: 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.365 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: _finish_auth 0 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 1 -- 172.21.15.6:0/55178601 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b800bb50 con 0x7fe6c8098670 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 5 monclient: authenticate success, global_id 4523 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:56.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 4 set_mon_vals no callback set 2023-09-30T00:46:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 1 -- 172.21.15.6:0/55178601 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b8011420 con 0x7fe6c8098670 2023-09-30T00:46:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:56.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6c77fe700 10 monclient: dump: 2023-09-30T00:46:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:56.369 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 10 monclient: shutdown 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:56.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 -- 172.21.15.6:0/55178601 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 msgr2=0x7fe6c8132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/55178601 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 0x7fe6c8132d00 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe6b8009910 tx=0x7fe6b80091a0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 -- 172.21.15.6:0/55178601 shutdown_connections 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/55178601 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098670 0x7fe6c8132d00 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 -- 172.21.15.6:0/55178601 >> 172.21.15.6:0/55178601 conn(0x7fe6c80a5480 msgr2=0x7fe6c80a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 -- 172.21.15.6:0/55178601 shutdown_connections 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.357+0000 7fe6ceb31700 1 -- 172.21.15.6:0/55178601 wait complete. 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.358+0000 7fe6ceb31700 10 monclient: build_initial_monmap 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.358+0000 7fe6ceb31700 10 monclient: monmap: 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:56.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.358+0000 7fe6ceb31700 1 Processor -- start 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.358+0000 7fe6ceb31700 1 -- start start 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.358+0000 7fe6ceb31700 10 monclient: init 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:56.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 0x7fe6c8141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): picked mon.noname-a con 0x7fe6c8098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8098a40 0x7fe6c812c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): picked mon.noname-b con 0x7fe6c8098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098e90 0x7fe6c812c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): picked mon.noname-c con 0x7fe6c8098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c8142740 con 0x7fe6c8098670 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c809e6e0 con 0x7fe6c8098a40 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:56.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6c812ed70 con 0x7fe6c8098e90 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6ceb31700 10 monclient(hunting): authenticate will time out at 1412.657495s 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 0x7fe6c8141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 0x7fe6c8141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46904/0 (socket says 172.21.15.6:46904) 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/18042455 learned_addr learned my addr 172.21.15.6:0/18042455 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): get_auth_request con 0x7fe6c8098670 auth_method 0 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098e90 0x7fe6c812c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:56.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): get_auth_request con 0x7fe6c8098e90 auth_method 0 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cd0ce700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient(hunting): handle_auth_done global_id 4521 payload 824 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6c7fff700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8098a40 0x7fe6c812c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098e90 msgr2=0x7fe6c812c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098e90 0x7fe6c812c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/18042455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8098a40 msgr2=0x7fe6c812c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8098a40 0x7fe6c812c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient: _finish_hunting 0 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 monclient: found mon.noname-a 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/18042455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6b8009420 con 0x7fe6c8098670 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6c7fff700 10 monclient: get_auth_request con 0x7fe6c8098a40 auth_method 0 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.359+0000 7fe6c7fff700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8098a40 0x7fe6c812c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6cc8cd700 1 -- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 msgr2=0x7fe6c8141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6cc8cd700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 0x7fe6c8141840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80030c0 tx=0x7fe6b8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:56.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b800bed0 con 0x7fe6c8098670 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: dump: 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:56.377 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: _finish_auth 0 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6b8020070 con 0x7fe6c8098670 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 5 monclient: authenticate success, global_id 4521 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 10 monclient: _renew_subs 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6c811c350 con 0x7fe6c8098670 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:56.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe6b801b6a0 con 0x7fe6c8098670 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c4ff9700 10 monclient: dump: 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:56.379 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 10 monclient: _renew_subs 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6c809f220 con 0x7fe6c8098670 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.362+0000 7fe6c6ffd700 4 set_mon_vals no callback set 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fe6b8022030 con 0x7fe6c8098670 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6ceb31700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6ceb31700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6ceb31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6c809f220 con 0x7fe6c8098670 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6c4ff9700 1 --2- 172.21.15.6:0/18042455 >> v2:172.21.15.22:6812/142603 conn(0x7fe6b40357a0 0x7fe6b4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:56.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fe6b8049730 con 0x7fe6c8098670 2023-09-30T00:46:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.363+0000 7fe6c7fff700 1 --2- 172.21.15.6:0/18042455 >> v2:172.21.15.22:6812/142603 conn(0x7fe6b40357a0 0x7fe6b4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.364+0000 7fe6c7fff700 10 monclient: get_auth_request con 0x7fe6b40357a0 auth_method 0 2023-09-30T00:46:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.366+0000 7fe6c7fff700 1 --2- 172.21.15.6:0/18042455 >> v2:172.21.15.22:6812/142603 conn(0x7fe6b40357a0 0x7fe6b4037c50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe6c812fbd0 tx=0x7fe6bc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.368+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fe6b801c450 con 0x7fe6c8098670 2023-09-30T00:46:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.368+0000 7fe6c4ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:56.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.368+0000 7fe6c4ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:46:56.618 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.617+0000 7fe6ceb31700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.617+0000 7fe6ceb31700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.617+0000 7fe6ceb31700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.617+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fe6c809f220 con 0x7fe6c8098670 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.618+0000 7fe6c4ff9700 1 -- 172.21.15.6:0/18042455 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fe6b801abc0 con 0x7fe6c8098670 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.618+0000 7fe6c4ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.618+0000 7fe6c4ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:46:56.619 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:46:56.620 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:46:56.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 >> v2:172.21.15.22:6812/142603 conn(0x7fe6b40357a0 msgr2=0x7fe6b4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/18042455 >> v2:172.21.15.22:6812/142603 conn(0x7fe6b40357a0 0x7fe6b4037c50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe6c812fbd0 tx=0x7fe6bc009040 comp rx=0 tx=0).stop 2023-09-30T00:46:56.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 10 monclient: shutdown 2023-09-30T00:46:56.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 msgr2=0x7fe6c8141840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 0x7fe6c8141840 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fe6b80030c0 tx=0x7fe6b8016000 comp rx=0 tx=0).stop 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 shutdown_connections 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/18042455 >> v2:172.21.15.22:6812/142603 conn(0x7fe6b40357a0 0x7fe6b4037c50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fe6c8098e90 0x7fe6c812c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fe6c8098670 0x7fe6c8141840 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 --2- 172.21.15.6:0/18042455 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fe6c8098a40 0x7fe6c812c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 >> 172.21.15.6:0/18042455 conn(0x7fe6c80a5480 msgr2=0x7fe6c812e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 shutdown_connections 2023-09-30T00:46:56.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:56.620+0000 7fe6ceb31700 1 -- 172.21.15.6:0/18042455 wait complete. 2023-09-30T00:46:59.635 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.773+0000 7f4c5d3ba700 10 monclient: get_monmap_and_config 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.773+0000 7f4c5d3ba700 10 monclient: build_initial_monmap 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.773+0000 7f4c5d3ba700 10 monclient: monmap: 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:46:59.774565+0000 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:46:59.774565+0000 2023-09-30T00:46:59.774 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:46:59.775 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.773+0000 7f4c5d3ba700 1 Processor -- start 2023-09-30T00:46:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.773+0000 7f4c5d3ba700 1 -- start start 2023-09-30T00:46:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient: init 2023-09-30T00:46:59.782 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4c5809e720 0x7f4c5809dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): picked mon.noname-a con 0x7f4c5809e720 addr v2:172.21.15.6:3300/0 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c5809eaf0 addr v1:172.21.15.6:6789/0 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c581329c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): picked mon.noname-b con 0x7f4c5809eec0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c58069490 con 0x7f4c5809eaf0 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c58069120 con 0x7f4c5809e720 2023-09-30T00:46:59.783 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c5809f2e0 con 0x7f4c5809eec0 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c5d3ba700 10 monclient(hunting): authenticate will time out at 1416.072560s 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c581329c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c56ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4c5809e720 0x7f4c5809dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c581329c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:32864/0 (socket says 172.21.15.6:32864) 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 1 -- 172.21.15.6:0/3042069127 learned_addr learned my addr 172.21.15.6:0/3042069127 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c56ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4c5809e720 0x7f4c5809dcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34460/0 (socket says 172.21.15.6:34460) 2023-09-30T00:46:59.784 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): get_auth_request con 0x7f4c5809eec0 auth_method 0 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c56ffd700 10 monclient(hunting): get_auth_request con 0x7f4c5809e720 auth_method 0 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c56ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c56ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c56ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:59.785 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.774+0000 7f4c577fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 10 monclient(hunting): handle_auth_done global_id 4529 payload 274 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 1 -- 172.21.15.6:0/3042069127 >> v2:172.21.15.6:3300/0 conn(0x7f4c5809e720 msgr2=0x7f4c5809dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 1 --2- 172.21.15.6:0/3042069127 >> v2:172.21.15.6:3300/0 conn(0x7f4c5809e720 0x7f4c5809dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 1 -- 172.21.15.6:0/3042069127 >> v1:172.21.15.6:6789/0 conn(0x7f4c5809eaf0 legacy=0x7f4c581322b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 1 monclient: found mon.noname-b 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.775+0000 7f4c577fe700 1 -- 172.21.15.6:0/3042069127 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c58069750 con 0x7f4c5809eec0 2023-09-30T00:46:59.786 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c577fe700 1 -- 172.21.15.6:0/3042069127 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 msgr2=0x7f4c581329c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c577fe700 1 --2- 172.21.15.6:0/3042069127 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c581329c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009f20 tx=0x7f4c50009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa4d3aa13f56116a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 1 -- 172.21.15.6:0/3042069127 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5000e070 con 0x7f4c5809eec0 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: dump: 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.787 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: _finish_auth 0 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 1 -- 172.21.15.6:0/3042069127 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5000bc50 con 0x7f4c5809eec0 2023-09-30T00:46:59.788 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 5 monclient: authenticate success, global_id 4529 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 10 monclient: get_monmap_and_config success 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 1 -- 172.21.15.6:0/3042069127 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c500092e0 con 0x7f4c5809eec0 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 4 set_mon_vals no callback set 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c55ffb700 10 monclient: dump: 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.789 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 10 monclient: shutdown 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.777+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/3042069127 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 msgr2=0x7f4c581329c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:59.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/3042069127 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c581329c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4c50009f20 tx=0x7f4c50009840 comp rx=0 tx=0).stop 2023-09-30T00:46:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/3042069127 shutdown_connections 2023-09-30T00:46:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/3042069127 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c581329c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/3042069127 >> v2:172.21.15.6:3300/0 conn(0x7f4c5809e720 0x7f4c5809dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/3042069127 >> 172.21.15.6:0/3042069127 conn(0x7f4c5809fdb0 msgr2=0x7f4c580a01b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:46:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/3042069127 shutdown_connections 2023-09-30T00:46:59.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/3042069127 wait complete. 2023-09-30T00:46:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 10 monclient: build_initial_monmap 2023-09-30T00:46:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.778+0000 7f4c5d3ba700 10 monclient: monmap: 2023-09-30T00:46:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:46:59.792 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:46:59.793 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 Processor -- start 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 -- start start 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient: init 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient: _reopen_session rank -1 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:46:59.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c5809e720 0x7f4c58134520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): picked mon.noname-b con 0x7f4c5809e720 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c5809eaf0 0x7f4c581416c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): picked mon.noname-a con 0x7f4c5809eaf0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c58141c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): picked mon.noname-c con 0x7f4c5809eec0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c581307d0 con 0x7f4c5809eaf0 2023-09-30T00:46:59.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58132750 con 0x7f4c5809e720 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c58132a30 con 0x7f4c5809eec0 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): _renew_subs 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c5d3ba700 10 monclient(hunting): authenticate will time out at 1416.077730s 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c5809eaf0 0x7f4c581416c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:59.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c58141c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c5809eaf0 0x7f4c581416c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34472/0 (socket says 172.21.15.6:34472) 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 1 -- 172.21.15.6:0/1100195861 learned_addr learned my addr 172.21.15.6:0/1100195861 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): get_auth_request con 0x7f4c5809eec0 auth_method 0 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:59.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 10 monclient(hunting): get_auth_request con 0x7f4c5809eaf0 auth_method 0 2023-09-30T00:46:59.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:59.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:59.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:59.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:59.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:59.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c577fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c567fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.779+0000 7f4c56ffd700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c5809e720 0x7f4c58134520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c567fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): get_auth_request con 0x7f4c5809e720 auth_method 0 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:46:59.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c56ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 10 monclient(hunting): handle_auth_done global_id 4532 payload 824 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 1 -- 172.21.15.6:0/1100195861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c5809e720 msgr2=0x7f4c58134520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c5809e720 0x7f4c58134520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:59.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 1 -- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c5809eaf0 msgr2=0x7f4c581416c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:46:59.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c5809eaf0 0x7f4c581416c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:46:59.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 10 monclient: _finish_hunting 0 2023-09-30T00:46:59.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 1 monclient: found mon.noname-c 2023-09-30T00:46:59.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:46:59.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:46:59.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.780+0000 7f4c577fe700 1 -- 172.21.15.6:0/1100195861 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c50009ae0 con 0x7f4c5809eec0 2023-09-30T00:46:59.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c577fe700 1 -- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 msgr2=0x7f4c58141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:46:59.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c577fe700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c58141c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c50014000 tx=0x7f4c50017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:59.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c50012b60 con 0x7f4c5809eec0 2023-09-30T00:46:59.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:59.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: dump: 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:59.805 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:59.806 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:59.806 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:59.806 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:59.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: _finish_auth 0 2023-09-30T00:46:59.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:46:59.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c50012cc0 con 0x7f4c5809eec0 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 5 monclient: authenticate success, global_id 4532 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5001e660 con 0x7f4c5809eec0 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 10 monclient: _renew_subs 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c58132d10 con 0x7f4c5809eec0 2023-09-30T00:46:59.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c557fa700 4 set_mon_vals no callback set 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c3effd700 10 monclient: dump: 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.808 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:46:59.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 10 monclient: _renew_subs 2023-09-30T00:46:59.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:59.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.782+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c58134a60 con 0x7f4c5809eec0 2023-09-30T00:46:59.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.783+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4c5001ea90 con 0x7f4c5809eec0 2023-09-30T00:46:59.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.783+0000 7f4c5d3ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:59.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.783+0000 7f4c5d3ba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:59.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.783+0000 7f4c5d3ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.783+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c5806f500 con 0x7f4c5809eec0 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.784+0000 7f4c3effd700 1 --2- 172.21.15.6:0/1100195861 >> v2:172.21.15.22:6812/142603 conn(0x7f4c440357a0 0x7f4c44037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.784+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4c50049350 con 0x7f4c5809eec0 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.784+0000 7f4c56ffd700 1 --2- 172.21.15.6:0/1100195861 >> v2:172.21.15.22:6812/142603 conn(0x7f4c440357a0 0x7f4c44037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.784+0000 7f4c56ffd700 10 monclient: get_auth_request con 0x7f4c440357a0 auth_method 0 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.787+0000 7f4c56ffd700 1 --2- 172.21.15.6:0/1100195861 >> v2:172.21.15.22:6812/142603 conn(0x7f4c440357a0 0x7f4c44037c50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4c48006aa0 tx=0x7f4c48008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:46:59.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.788+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c50012e30 con 0x7f4c5809eec0 2023-09-30T00:46:59.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.788+0000 7f4c3effd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:46:59.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:46:59.788+0000 7f4c3effd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c5d3ba700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c5d3ba700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c5d3ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4c58132b70 con 0x7f4c5809eec0 2023-09-30T00:47:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c3effd700 1 -- 172.21.15.6:0/1100195861 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4c58132b70 con 0x7f4c5809eec0 2023-09-30T00:47:00.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c3effd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:00.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.034+0000 7f4c3effd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:00.036 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:00.036 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 >> v2:172.21.15.22:6812/142603 conn(0x7f4c440357a0 msgr2=0x7f4c44037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/1100195861 >> v2:172.21.15.22:6812/142603 conn(0x7f4c440357a0 0x7f4c44037c50 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4c48006aa0 tx=0x7f4c48008040 comp rx=0 tx=0).stop 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 10 monclient: shutdown 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 msgr2=0x7f4c58141c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c58141c00 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4c50014000 tx=0x7f4c50017000 comp rx=0 tx=0).stop 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 shutdown_connections 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/1100195861 >> v2:172.21.15.22:6812/142603 conn(0x7f4c440357a0 0x7f4c44037c50 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c5809eec0 0x7f4c58141c00 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:00.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c5809e720 0x7f4c58134520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:00.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 --2- 172.21.15.6:0/1100195861 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c5809eaf0 0x7f4c581416c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:00.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 >> 172.21.15.6:0/1100195861 conn(0x7f4c5809fdb0 msgr2=0x7f4c5806e480 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:00.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 shutdown_connections 2023-09-30T00:47:00.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:00.037+0000 7f4c5d3ba700 1 -- 172.21.15.6:0/1100195861 wait complete. 2023-09-30T00:47:03.054 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.191+0000 7fc8654be700 10 monclient: get_monmap_and_config 2023-09-30T00:47:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.191+0000 7fc8654be700 10 monclient: build_initial_monmap 2023-09-30T00:47:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.191+0000 7fc8654be700 10 monclient: monmap: 2023-09-30T00:47:03.192 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:03.192889+0000 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:03.192889+0000 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:03.193 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:03.194 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 Processor -- start 2023-09-30T00:47:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 -- start start 2023-09-30T00:47:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient: init 2023-09-30T00:47:03.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient: _add_conns ranks=[4,3,1,2,0] 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc860098550 0x7fc86012f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): picked mon.noname-c con 0x7fc860098550 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc860098920 0x7fc8601321d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): picked mon.noname-a con 0x7fc860098920 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc860098dc0 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc860096d60 con 0x7fc860098dc0 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8600a1c40 con 0x7fc860098920 2023-09-30T00:47:03.196 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc860097040 con 0x7fc860098550 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc8654be700 10 monclient(hunting): authenticate will time out at 1419.490869s 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.192+0000 7fc85e7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc860098920 0x7fc8601321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc860098920 0x7fc8601321d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34480/0 (socket says 172.21.15.6:34480) 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 -- 172.21.15.6:0/817027544 learned_addr learned my addr 172.21.15.6:0/817027544 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): get_auth_request con 0x7fc860098920 auth_method 0 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85effd700 1 --2- 172.21.15.6:0/817027544 >> v2:172.21.15.22:3300/0 conn(0x7fc860098550 0x7fc86012f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_done global_id 4530 payload 274 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 -- 172.21.15.6:0/817027544 >> v2:172.21.15.22:3300/0 conn(0x7fc860098550 msgr2=0x7fc86012f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 --2- 172.21.15.6:0/817027544 >> v2:172.21.15.22:3300/0 conn(0x7fc860098550 0x7fc86012f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 -- 172.21.15.6:0/817027544 >> v1:172.21.15.22:6789/0 conn(0x7fc860098dc0 legacy=0x7fc860132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 monclient: found mon.noname-a 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:03.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.193+0000 7fc85e7fc700 1 -- 172.21.15.6:0/817027544 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc860068df0 con 0x7fc860098920 2023-09-30T00:47:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85e7fc700 1 -- 172.21.15.6:0/817027544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 msgr2=0x7fc8601321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85e7fc700 1 --2- 172.21.15.6:0/817027544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc8601321d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc86006a650 tx=0x7fc854009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ebd643c4d0a502c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 1 -- 172.21.15.6:0/817027544 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85400ba30 con 0x7fc860098920 2023-09-30T00:47:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:03.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: dump: 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:03.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc8654be700 5 monclient: authenticate success, global_id 4530 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc8654be700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 1 -- 172.21.15.6:0/817027544 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc85400bb90 con 0x7fc860098920 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc8654be700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc8654be700 4 set_mon_vals no callback set 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 1 -- 172.21.15.6:0/817027544 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854011420 con 0x7fc860098920 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.195+0000 7fc85dffb700 10 monclient: dump: 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:03.206 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 10 monclient: shutdown 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 -- 172.21.15.6:0/817027544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 msgr2=0x7fc8601321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 --2- 172.21.15.6:0/817027544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc8601321d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc86006a650 tx=0x7fc854009250 comp rx=0 tx=0).stop 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 -- 172.21.15.6:0/817027544 shutdown_connections 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 --2- 172.21.15.6:0/817027544 >> v2:172.21.15.22:3300/0 conn(0x7fc860098550 0x7fc86012f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 --2- 172.21.15.6:0/817027544 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc8601321d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 -- 172.21.15.6:0/817027544 >> 172.21.15.6:0/817027544 conn(0x7fc8600a5480 msgr2=0x7fc8600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 -- 172.21.15.6:0/817027544 shutdown_connections 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.196+0000 7fc8654be700 1 -- 172.21.15.6:0/817027544 wait complete. 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 10 monclient: build_initial_monmap 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 10 monclient: monmap: 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:03.208 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 1 Processor -- start 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 1 -- start start 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 10 monclient: init 2023-09-30T00:47:03.209 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.197+0000 7fc8654be700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc860098550 0x7fc860131fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): picked mon.noname-c con 0x7fc860098550 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc86012ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): picked mon.noname-a con 0x7fc860098920 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc860098dc0 0x7fc860099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): picked mon.noname-b con 0x7fc860098dc0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8600a3dd0 con 0x7fc860098920 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.210 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc860141db0 con 0x7fc860098dc0 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc86009b970 con 0x7fc860098550 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc8654be700 10 monclient(hunting): authenticate will time out at 1419.496073s 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc86012ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc860098550 0x7fc860131fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc86012ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34484/0 (socket says 172.21.15.6:34484) 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 -- 172.21.15.6:0/3675685417 learned_addr learned my addr 172.21.15.6:0/3675685417 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): get_auth_request con 0x7fc860098920 auth_method 0 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): get_auth_request con 0x7fc860098550 auth_method 0 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:03.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85f7fe700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc860098dc0 0x7fc860099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient(hunting): handle_auth_done global_id 4533 payload 824 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 -- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc860098550 msgr2=0x7fc860131fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc860098550 0x7fc860131fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 -- 172.21.15.6:0/3675685417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc860098dc0 msgr2=0x7fc860099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc860098dc0 0x7fc860099290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 monclient: found mon.noname-a 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85e7fc700 1 -- 172.21.15.6:0/3675685417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8540094f0 con 0x7fc860098920 2023-09-30T00:47:03.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85f7fe700 10 monclient: get_auth_request con 0x7fc860098dc0 auth_method 0 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.198+0000 7fc85f7fe700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc860098dc0 0x7fc860099290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc85e7fc700 1 -- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 msgr2=0x7fc86012ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc85e7fc700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc86012ded0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc86006a680 tx=0x7fc854016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc854006c00 con 0x7fc860098920 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 10 monclient: dump: 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.214 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc8654be700 5 monclient: authenticate success, global_id 4533 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.200+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc854020070 con 0x7fc860098920 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc8654be700 10 monclient: _renew_subs 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc8654be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:03.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc86009bc50 con 0x7fc860098920 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc846ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc85401b680 con 0x7fc860098920 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc846ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc846ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc846ffd700 10 monclient: dump: 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:03.216 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc85d7fa700 4 set_mon_vals no callback set 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc8654be700 10 monclient: _renew_subs 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc8654be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc860142620 con 0x7fc860098920 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.201+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc854022030 con 0x7fc860098920 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc846ffd700 1 --2- 172.21.15.6:0/3675685417 >> v2:172.21.15.22:6812/142603 conn(0x7fc84c0357a0 0x7fc84c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc854049850 con 0x7fc860098920 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc8654be700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc8654be700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc8654be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:03.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc860072b00 con 0x7fc860098920 2023-09-30T00:47:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc85effd700 1 --2- 172.21.15.6:0/3675685417 >> v2:172.21.15.22:6812/142603 conn(0x7fc84c0357a0 0x7fc84c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.202+0000 7fc85effd700 10 monclient: get_auth_request con 0x7fc84c0357a0 auth_method 0 2023-09-30T00:47:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.205+0000 7fc85effd700 1 --2- 172.21.15.6:0/3675685417 >> v2:172.21.15.22:6812/142603 conn(0x7fc84c0357a0 0x7fc84c037c50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc85000ac20 tx=0x7fc850008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.207+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc85401c970 con 0x7fc860098920 2023-09-30T00:47:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.207+0000 7fc846ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:03.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.207+0000 7fc846ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc8654be700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc8654be700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc8654be700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc860072b00 con 0x7fc860098920 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc846ffd700 1 -- 172.21.15.6:0/3675685417 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc854024b80 con 0x7fc860098920 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc846ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.455+0000 7fc846ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:03.456 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:03.457 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:03.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 >> v2:172.21.15.22:6812/142603 conn(0x7fc84c0357a0 msgr2=0x7fc84c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 --2- 172.21.15.6:0/3675685417 >> v2:172.21.15.22:6812/142603 conn(0x7fc84c0357a0 0x7fc84c037c50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fc85000ac20 tx=0x7fc850008040 comp rx=0 tx=0).stop 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 10 monclient: shutdown 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 msgr2=0x7fc86012ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc86012ded0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fc86006a680 tx=0x7fc854016000 comp rx=0 tx=0).stop 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 shutdown_connections 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 --2- 172.21.15.6:0/3675685417 >> v2:172.21.15.22:6812/142603 conn(0x7fc84c0357a0 0x7fc84c037c50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc860098dc0 0x7fc860099290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc860098550 0x7fc860131fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 --2- 172.21.15.6:0/3675685417 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc860098920 0x7fc86012ded0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 >> 172.21.15.6:0/3675685417 conn(0x7fc8600a5480 msgr2=0x7fc86009e7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.457+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 shutdown_connections 2023-09-30T00:47:03.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:03.458+0000 7fc8654be700 1 -- 172.21.15.6:0/3675685417 wait complete. 2023-09-30T00:47:06.472 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.610+0000 7f4a25a77700 10 monclient: get_monmap_and_config 2023-09-30T00:47:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.610+0000 7f4a25a77700 10 monclient: build_initial_monmap 2023-09-30T00:47:06.611 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.610+0000 7f4a25a77700 10 monclient: monmap: 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:06.612177+0000 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:06.612177+0000 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:06.612 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:06.614 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.611+0000 7f4a25a77700 1 Processor -- start 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.611+0000 7f4a25a77700 1 -- start start 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.611+0000 7f4a25a77700 10 monclient: init 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient: _add_conns ranks=[1,2,0,4,3] 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4a20098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 0x7f4a20137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient(hunting): picked mon.noname-b con 0x7f4a20132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4a20137b60 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200a1000 con 0x7f4a20137b60 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a200a0730 con 0x7f4a20098670 2023-09-30T00:47:06.615 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a20099150 con 0x7f4a20132d00 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a25a77700 10 monclient(hunting): authenticate will time out at 1422.910369s 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 0x7f4a20137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1ffff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4a20137b60 0x7f4a20139f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45652/0 (socket says 172.21.15.6:45652) 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1ffff700 1 -- 172.21.15.6:0/1727444973 learned_addr learned my addr 172.21.15.6:0/1727444973 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): get_auth_request con 0x7f4a20132d00 auth_method 0 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:06.616 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient(hunting): handle_auth_done global_id 4538 payload 274 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 1 -- 172.21.15.6:0/1727444973 >> v1:172.21.15.22:6789/0 conn(0x7f4a20098670 legacy=0x7f4a20098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 1 -- 172.21.15.6:0/1727444973 >> v1:172.21.15.6:6789/0 conn(0x7f4a20137b60 legacy=0x7f4a20139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 1 monclient: found mon.noname-b 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:06.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.612+0000 7f4a1effd700 1 -- 172.21.15.6:0/1727444973 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a20068df0 con 0x7f4a20132d00 2023-09-30T00:47:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1effd700 1 -- 172.21.15.6:0/1727444973 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 msgr2=0x7f4a20137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1effd700 1 --2- 172.21.15.6:0/1727444973 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 0x7f4a20137620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009910 tx=0x7f4a140091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df8e512672ad9e6e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 1 -- 172.21.15.6:0/1727444973 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a1400b9f0 con 0x7f4a20132d00 2023-09-30T00:47:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:06.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: dump: 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:06.624 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 1 -- 172.21.15.6:0/1727444973 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a1400bb50 con 0x7f4a20132d00 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a25a77700 5 monclient: authenticate success, global_id 4538 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a25a77700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a25a77700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 1 -- 172.21.15.6:0/1727444973 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a14011420 con 0x7f4a20132d00 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a25a77700 4 set_mon_vals no callback set 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.615+0000 7f4a1e7fc700 10 monclient: dump: 2023-09-30T00:47:06.625 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 10 monclient: shutdown 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:06.626 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 -- 172.21.15.6:0/1727444973 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 msgr2=0x7f4a20137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 --2- 172.21.15.6:0/1727444973 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 0x7f4a20137620 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a14009910 tx=0x7f4a140091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 -- 172.21.15.6:0/1727444973 shutdown_connections 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 --2- 172.21.15.6:0/1727444973 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20132d00 0x7f4a20137620 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 -- 172.21.15.6:0/1727444973 >> 172.21.15.6:0/1727444973 conn(0x7f4a200a5480 msgr2=0x7f4a200a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 -- 172.21.15.6:0/1727444973 shutdown_connections 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 1 -- 172.21.15.6:0/1727444973 wait complete. 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 10 monclient: build_initial_monmap 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.616+0000 7f4a25a77700 10 monclient: monmap: 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:06.627 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 Processor -- start 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 -- start start 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient: init 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:06.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 0x7f4a2012f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): picked mon.noname-a con 0x7f4a20098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a20132d00 0x7f4a2009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): picked mon.noname-b con 0x7f4a20132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20137b60 0x7f4a2009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): picked mon.noname-c con 0x7f4a20137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2009dce0 con 0x7f4a20098670 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a200a1570 con 0x7f4a20132d00 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:06.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4a2009e930 con 0x7f4a20137b60 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a25a77700 10 monclient(hunting): authenticate will time out at 1422.915854s 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 0x7f4a2012f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 0x7f4a2012f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51688/0 (socket says 172.21.15.6:51688) 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 1 -- 172.21.15.6:0/185106511 learned_addr learned my addr 172.21.15.6:0/185106511 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 10 monclient(hunting): get_auth_request con 0x7f4a20098670 auth_method 0 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.617+0000 7f4a1ffff700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20137b60 0x7f4a2009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): get_auth_request con 0x7f4a20137b60 auth_method 0 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a20132d00 0x7f4a2009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): get_auth_request con 0x7f4a20132d00 auth_method 0 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 10 monclient(hunting): handle_auth_done global_id 4536 payload 824 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 1 -- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20137b60 msgr2=0x7f4a2009cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20137b60 0x7f4a2009cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 1 -- 172.21.15.6:0/185106511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a20132d00 msgr2=0x7f4a2009c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a20132d00 0x7f4a2009c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 1 monclient: found mon.noname-a 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1f7fe700 1 -- 172.21.15.6:0/185106511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a14009420 con 0x7f4a20098670 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.618+0000 7f4a1ffff700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20137b60 0x7f4a2009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a1f7fe700 1 -- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 msgr2=0x7f4a2012f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a1f7fe700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 0x7f4a2012f670 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000c5d0 tx=0x7f4a10007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a100117a0 con 0x7f4a20098670 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: dump: 2023-09-30T00:47:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4a10011900 con 0x7f4a20098670 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 5 monclient: authenticate success, global_id 4536 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 10 monclient: _renew_subs 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a20133410 con 0x7f4a20098670 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a10011cc0 con 0x7f4a20098670 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a077fe700 10 monclient: dump: 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:06.635 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a1dffb700 4 set_mon_vals no callback set 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 10 monclient: _renew_subs 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.620+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a200a33e0 con 0x7f4a20098670 2023-09-30T00:47:06.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.621+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4a10012630 con 0x7f4a20098670 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.621+0000 7f4a25a77700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.621+0000 7f4a25a77700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.621+0000 7f4a25a77700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.621+0000 7f4a077fe700 1 --2- 172.21.15.6:0/185106511 >> v2:172.21.15.22:6812/142603 conn(0x7f4a0c0357a0 0x7f4a0c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.621+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a20072b00 con 0x7f4a20098670 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.622+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4a10047630 con 0x7f4a20098670 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.622+0000 7f4a1effd700 1 --2- 172.21.15.6:0/185106511 >> v2:172.21.15.22:6812/142603 conn(0x7f4a0c0357a0 0x7f4a0c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:06.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.622+0000 7f4a1effd700 10 monclient: get_auth_request con 0x7f4a0c0357a0 auth_method 0 2023-09-30T00:47:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.624+0000 7f4a1effd700 1 --2- 172.21.15.6:0/185106511 >> v2:172.21.15.22:6812/142603 conn(0x7f4a0c0357a0 0x7f4a0c037c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a140030c0 tx=0x7f4a14016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.627+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4a10011e60 con 0x7f4a20098670 2023-09-30T00:47:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.627+0000 7f4a077fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:06.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.627+0000 7f4a077fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a25a77700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a25a77700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a25a77700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4a20133750 con 0x7f4a20098670 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a077fe700 1 -- 172.21.15.6:0/185106511 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4a100102e0 con 0x7f4a20098670 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a077fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.875+0000 7f4a077fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:06.876 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:06.877 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:06.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 >> v2:172.21.15.22:6812/142603 conn(0x7f4a0c0357a0 msgr2=0x7f4a0c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:06.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 --2- 172.21.15.6:0/185106511 >> v2:172.21.15.22:6812/142603 conn(0x7f4a0c0357a0 0x7f4a0c037c50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4a140030c0 tx=0x7f4a14016000 comp rx=0 tx=0).stop 2023-09-30T00:47:06.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 10 monclient: shutdown 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 msgr2=0x7f4a2012f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 0x7f4a2012f670 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f4a1000c5d0 tx=0x7f4a10007f60 comp rx=0 tx=0).stop 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 shutdown_connections 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 --2- 172.21.15.6:0/185106511 >> v2:172.21.15.22:6812/142603 conn(0x7f4a0c0357a0 0x7f4a0c037c50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4a20137b60 0x7f4a2009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4a20098670 0x7f4a2012f670 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 --2- 172.21.15.6:0/185106511 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4a20132d00 0x7f4a2009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 >> 172.21.15.6:0/185106511 conn(0x7f4a200a5480 msgr2=0x7f4a200a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 shutdown_connections 2023-09-30T00:47:06.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:06.877+0000 7f4a25a77700 1 -- 172.21.15.6:0/185106511 wait complete. 2023-09-30T00:47:09.892 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:10.029 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.029+0000 7f6751126700 10 monclient: get_monmap_and_config 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.029+0000 7f6751126700 10 monclient: build_initial_monmap 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.029+0000 7f6751126700 10 monclient: monmap: 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:10.030404+0000 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:10.030404+0000 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:10.030 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:10.031 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:10.031 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.029+0000 7f6751126700 1 Processor -- start 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.029+0000 7f6751126700 1 -- start start 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.029+0000 7f6751126700 10 monclient: init 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient: _add_conns ranks=[0,1,2,4,3] 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f674c098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f674c132d00 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 0x7f674c13b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient(hunting): picked mon.noname-b con 0x7f674c138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:10.035 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c0a1000 con 0x7f674c098670 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f674c0a0730 con 0x7f674c132d00 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c099150 con 0x7f674c138d40 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f6751126700 10 monclient(hunting): authenticate will time out at 1426.328400s 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674ad9d700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f674c098670 0x7f674c098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:45664/0 (socket says 172.21.15.6:45664) 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674ad9d700 1 -- 172.21.15.6:0/3174632456 learned_addr learned my addr 172.21.15.6:0/3174632456 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 1 --2- 172.21.15.6:0/3174632456 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 0x7f674c13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): get_auth_request con 0x7f674c138d40 auth_method 0 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:10.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient(hunting): handle_auth_done global_id 4544 payload 274 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 1 -- 172.21.15.6:0/3174632456 >> v1:172.21.15.22:6789/0 conn(0x7f674c132d00 legacy=0x7f674c137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 1 -- 172.21.15.6:0/3174632456 >> v1:172.21.15.6:6789/0 conn(0x7f674c098670 legacy=0x7f674c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient: _finish_hunting 0 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 1 monclient: found mon.noname-b 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.030+0000 7f674b59e700 1 -- 172.21.15.6:0/3174632456 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f674c068df0 con 0x7f674c138d40 2023-09-30T00:47:10.037 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f674b59e700 1 -- 172.21.15.6:0/3174632456 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 msgr2=0x7f674c13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f674b59e700 1 --2- 172.21.15.6:0/3174632456 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 0x7f674c13b120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6744009e90 tx=0x7f6744009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc89a982718b95ce server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 1 -- 172.21.15.6:0/3174632456 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674400e070 con 0x7f674c138d40 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: dump: 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:10.038 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: _finish_auth 0 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 5 monclient: authenticate success, global_id 4544 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 1 -- 172.21.15.6:0/3174632456 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6744012430 con 0x7f674c138d40 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 1 -- 172.21.15.6:0/3174632456 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f674400bea0 con 0x7f674c138d40 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:10.039 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6749d9b700 10 monclient: dump: 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:10.040 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 4 set_mon_vals no callback set 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 10 monclient: shutdown 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 1 -- 172.21.15.6:0/3174632456 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 msgr2=0x7f674c13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 1 --2- 172.21.15.6:0/3174632456 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 0x7f674c13b120 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f6744009e90 tx=0x7f6744009780 comp rx=0 tx=0).stop 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 1 -- 172.21.15.6:0/3174632456 shutdown_connections 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 1 --2- 172.21.15.6:0/3174632456 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c138d40 0x7f674c13b120 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.033+0000 7f6751126700 1 -- 172.21.15.6:0/3174632456 >> 172.21.15.6:0/3174632456 conn(0x7f674c0a5480 msgr2=0x7f674c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.034+0000 7f6751126700 1 -- 172.21.15.6:0/3174632456 shutdown_connections 2023-09-30T00:47:10.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.034+0000 7f6751126700 1 -- 172.21.15.6:0/3174632456 wait complete. 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.034+0000 7f6751126700 10 monclient: build_initial_monmap 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.034+0000 7f6751126700 10 monclient: monmap: 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:10.042 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:10.043 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:10.043 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 Processor -- start 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 -- start start 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient: init 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 0x7f674c138410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:10.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): picked mon.noname-c con 0x7f674c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f674c132d00 0x7f674c141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): picked mon.noname-b con 0x7f674c132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 0x7f674c09c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): picked mon.noname-a con 0x7f674c138d40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c133c80 con 0x7f674c138d40 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f674c130320 con 0x7f674c132d00 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f674c133630 con 0x7f674c098670 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:10.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f6751126700 10 monclient(hunting): authenticate will time out at 1426.333589s 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 0x7f674c138410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 0x7f674c138410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:52822/0 (socket says 172.21.15.6:52822) 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 1 -- 172.21.15.6:0/548636983 learned_addr learned my addr 172.21.15.6:0/548636983 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): get_auth_request con 0x7f674c098670 auth_method 0 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 0x7f674c09c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): get_auth_request con 0x7f674c138d40 auth_method 0 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:10.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674a59c700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f674c132d00 0x7f674c141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674a59c700 10 monclient(hunting): get_auth_request con 0x7f674c132d00 auth_method 0 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.035+0000 7f674b59e700 10 monclient(hunting): handle_auth_done global_id 4539 payload 824 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 1 -- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 msgr2=0x7f674c138410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 0x7f674c138410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 1 -- 172.21.15.6:0/548636983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f674c132d00 msgr2=0x7f674c141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f674c132d00 0x7f674c141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 10 monclient: _finish_hunting 0 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 1 monclient: found mon.noname-a 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674b59e700 1 -- 172.21.15.6:0/548636983 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744009a20 con 0x7f674c138d40 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.036+0000 7f674ad9d700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 0x7f674c138410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f674b59e700 1 -- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 msgr2=0x7f674c09c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f674b59e700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 0x7f674c09c7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6744009fd0 tx=0x7f6744018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744026430 con 0x7f674c138d40 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: dump: 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:10.052 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 5 monclient: authenticate success, global_id 4539 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 10 monclient: _renew_subs 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6744021070 con 0x7f674c138d40 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f674c133910 con 0x7f674c138d40 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6744026800 con 0x7f674c138d40 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6732ffd700 10 monclient: dump: 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f674959a700 4 set_mon_vals no callback set 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 10 monclient: _renew_subs 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.038+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f674c09e120 con 0x7f674c138d40 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6744023030 con 0x7f674c138d40 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6751126700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6751126700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6751126700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f674c072b00 con 0x7f674c138d40 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6732ffd700 1 --2- 172.21.15.6:0/548636983 >> v2:172.21.15.22:6812/142603 conn(0x7f67380357a0 0x7f6738037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f674404a1f0 con 0x7f674c138d40 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f674ad9d700 1 --2- 172.21.15.6:0/548636983 >> v2:172.21.15.22:6812/142603 conn(0x7f67380357a0 0x7f6738037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:10.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.039+0000 7f674ad9d700 10 monclient: get_auth_request con 0x7f67380357a0 auth_method 0 2023-09-30T00:47:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.042+0000 7f674ad9d700 1 --2- 172.21.15.6:0/548636983 >> v2:172.21.15.22:6812/142603 conn(0x7f67380357a0 0x7f6738037c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f673c0052e0 tx=0x7f673c004a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.044+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f67440117a0 con 0x7f674c138d40 2023-09-30T00:47:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.044+0000 7f6732ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:10.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.044+0000 7f6732ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:10.294 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.293+0000 7f6751126700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.293+0000 7f6751126700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.293+0000 7f6751126700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.293+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f674c072b00 con 0x7f674c138d40 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.294+0000 7f6732ffd700 1 -- 172.21.15.6:0/548636983 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f67440253d0 con 0x7f674c138d40 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.294+0000 7f6732ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.294+0000 7f6732ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:10.295 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 >> v2:172.21.15.22:6812/142603 conn(0x7f67380357a0 msgr2=0x7f6738037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 --2- 172.21.15.6:0/548636983 >> v2:172.21.15.22:6812/142603 conn(0x7f67380357a0 0x7f6738037c50 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f673c0052e0 tx=0x7f673c004a80 comp rx=0 tx=0).stop 2023-09-30T00:47:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 10 monclient: shutdown 2023-09-30T00:47:10.297 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 msgr2=0x7f674c09c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 0x7f674c09c7a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6744009fd0 tx=0x7f6744018000 comp rx=0 tx=0).stop 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 shutdown_connections 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 --2- 172.21.15.6:0/548636983 >> v2:172.21.15.22:6812/142603 conn(0x7f67380357a0 0x7f6738037c50 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f674c138d40 0x7f674c09c7a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f674c098670 0x7f674c138410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 --2- 172.21.15.6:0/548636983 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f674c132d00 0x7f674c141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 >> 172.21.15.6:0/548636983 conn(0x7f674c0a5480 msgr2=0x7f674c0975f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 shutdown_connections 2023-09-30T00:47:10.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:10.296+0000 7f6751126700 1 -- 172.21.15.6:0/548636983 wait complete. 2023-09-30T00:47:13.311 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.449+0000 7f8f99792700 10 monclient: get_monmap_and_config 2023-09-30T00:47:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.449+0000 7f8f99792700 10 monclient: build_initial_monmap 2023-09-30T00:47:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.449+0000 7f8f99792700 10 monclient: monmap: 2023-09-30T00:47:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:13.450 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:13.450859+0000 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:13.450859+0000 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:13.451 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 Processor -- start 2023-09-30T00:47:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 -- start start 2023-09-30T00:47:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient: init 2023-09-30T00:47:13.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient: _add_conns ranks=[3,4,1,2,0] 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f94098550 0x7f8f9412f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): picked mon.noname-a con 0x7f8f94098550 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8f94098920 0x7f8f941321d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): picked mon.noname-c con 0x7f8f94098920 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8f94098dc0 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f94096d60 con 0x7f8f94098dc0 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f940a1c40 con 0x7f8f94098550 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f94097040 con 0x7f8f94098920 2023-09-30T00:47:13.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.450+0000 7f8f99792700 10 monclient(hunting): authenticate will time out at 1429.748955s 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f94098550 0x7f8f9412f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8f94098550 0x7f8f9412f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51700/0 (socket says 172.21.15.6:51700) 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 -- 172.21.15.6:0/375964437 learned_addr learned my addr 172.21.15.6:0/375964437 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): get_auth_request con 0x7f8f94098550 auth_method 0 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f927fc700 1 --2- 172.21.15.6:0/375964437 >> v2:172.21.15.22:3300/0 conn(0x7f8f94098920 0x7f8f941321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:13.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f927fc700 10 monclient(hunting): get_auth_request con 0x7f8f94098920 auth_method 0 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f927fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f927fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_done global_id 4542 payload 274 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 -- 172.21.15.6:0/375964437 >> v2:172.21.15.22:3300/0 conn(0x7f8f94098920 msgr2=0x7f8f941321d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 --2- 172.21.15.6:0/375964437 >> v2:172.21.15.22:3300/0 conn(0x7f8f94098920 0x7f8f941321d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 -- 172.21.15.6:0/375964437 >> v1:172.21.15.22:6789/0 conn(0x7f8f94098dc0 legacy=0x7f8f94132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 monclient: found mon.noname-a 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:13.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.451+0000 7f8f92ffd700 1 -- 172.21.15.6:0/375964437 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94068df0 con 0x7f8f94098550 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f92ffd700 1 -- 172.21.15.6:0/375964437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098550 msgr2=0x7f8f9412f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f92ffd700 1 --2- 172.21.15.6:0/375964437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098550 0x7f8f9412f760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8f940a1b40 tx=0x7f8f88009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c78fa3da3066ec8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 1 -- 172.21.15.6:0/375964437 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f8800ba30 con 0x7f8f94098550 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: dump: 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:13.459 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f99792700 5 monclient: authenticate success, global_id 4542 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 1 -- 172.21.15.6:0/375964437 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8800bb90 con 0x7f8f94098550 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f99792700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f99792700 4 set_mon_vals no callback set 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 1 -- 172.21.15.6:0/375964437 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f88011420 con 0x7f8f94098550 2023-09-30T00:47:13.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.453+0000 7f8f91ffb700 10 monclient: dump: 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:13.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 10 monclient: shutdown 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 -- 172.21.15.6:0/375964437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098550 msgr2=0x7f8f9412f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 --2- 172.21.15.6:0/375964437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098550 0x7f8f9412f760 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f8f940a1b40 tx=0x7f8f88009250 comp rx=0 tx=0).stop 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 -- 172.21.15.6:0/375964437 shutdown_connections 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 --2- 172.21.15.6:0/375964437 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098550 0x7f8f9412f760 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 --2- 172.21.15.6:0/375964437 >> v2:172.21.15.22:3300/0 conn(0x7f8f94098920 0x7f8f941321d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 -- 172.21.15.6:0/375964437 >> 172.21.15.6:0/375964437 conn(0x7f8f940a5480 msgr2=0x7f8f940a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 -- 172.21.15.6:0/375964437 shutdown_connections 2023-09-30T00:47:13.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.454+0000 7f8f99792700 1 -- 172.21.15.6:0/375964437 wait complete. 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.455+0000 7f8f99792700 10 monclient: build_initial_monmap 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.455+0000 7f8f99792700 10 monclient: monmap: 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:13.467 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.455+0000 7f8f99792700 1 Processor -- start 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 -- start start 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient: init 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8f94098550 0x7f8f9409e820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): picked mon.noname-c con 0x7f8f94098550 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 0x7f8f9412dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): picked mon.noname-a con 0x7f8f94098920 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f94098dc0 0x7f8f9412fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:13.468 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): picked mon.noname-b con 0x7f8f94098dc0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f94132440 con 0x7f8f94098920 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f94132970 con 0x7f8f94098dc0 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f940a1230 con 0x7f8f94098550 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f99792700 10 monclient(hunting): authenticate will time out at 1429.754814s 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 0x7f8f9412dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 0x7f8f9412dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51708/0 (socket says 172.21.15.6:51708) 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f92ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8f94098550 0x7f8f9409e820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 1 -- 172.21.15.6:0/3812283120 learned_addr learned my addr 172.21.15.6:0/3812283120 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:13.469 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 10 monclient(hunting): get_auth_request con 0x7f8f94098920 auth_method 0 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f927fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f92ffd700 10 monclient(hunting): get_auth_request con 0x7f8f94098550 auth_method 0 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f92ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f92ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.456+0000 7f8f92ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:13.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f92ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 10 monclient(hunting): handle_auth_done global_id 4545 payload 824 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f937fe700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f94098dc0 0x7f8f9412fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 1 -- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8f94098550 msgr2=0x7f8f9409e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8f94098550 0x7f8f9409e820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 1 -- 172.21.15.6:0/3812283120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f94098dc0 msgr2=0x7f8f9412fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f94098dc0 0x7f8f9412fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 1 monclient: found mon.noname-a 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f927fc700 1 -- 172.21.15.6:0/3812283120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f880094f0 con 0x7f8f94098920 2023-09-30T00:47:13.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f937fe700 10 monclient: get_auth_request con 0x7f8f94098dc0 auth_method 0 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.457+0000 7f8f937fe700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f94098dc0 0x7f8f9412fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f927fc700 1 -- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 msgr2=0x7f8f9412dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f927fc700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 0x7f8f9412dd80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400d890 tx=0x7f8f84008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84012800 con 0x7f8f94098920 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: dump: 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.472 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: _finish_auth 0 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f84012960 con 0x7f8f94098920 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f84012ce0 con 0x7f8f94098920 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 5 monclient: authenticate success, global_id 4545 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:13.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f7affd700 10 monclient: dump: 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 10 monclient: _renew_subs 2023-09-30T00:47:13.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f917fa700 4 set_mon_vals no callback set 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f94130d70 con 0x7f8f94098920 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 10 monclient: _renew_subs 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.459+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f94131450 con 0x7f8f94098920 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f99792700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8f84014640 con 0x7f8f94098920 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f99792700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f99792700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f94072b00 con 0x7f8f94098920 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f7affd700 1 --2- 172.21.15.6:0/3812283120 >> v2:172.21.15.22:6812/142603 conn(0x7f8f800357a0 0x7f8f80037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:13.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.460+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8f84047fa0 con 0x7f8f94098920 2023-09-30T00:47:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.461+0000 7f8f92ffd700 1 --2- 172.21.15.6:0/3812283120 >> v2:172.21.15.22:6812/142603 conn(0x7f8f800357a0 0x7f8f80037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.461+0000 7f8f92ffd700 10 monclient: get_auth_request con 0x7f8f800357a0 auth_method 0 2023-09-30T00:47:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.463+0000 7f8f92ffd700 1 --2- 172.21.15.6:0/3812283120 >> v2:172.21.15.22:6812/142603 conn(0x7f8f800357a0 0x7f8f80037c50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8f940a1b40 tx=0x7f8f88016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.465+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8f84012e40 con 0x7f8f94098920 2023-09-30T00:47:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.465+0000 7f8f7affd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:13.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.465+0000 7f8f7affd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f99792700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f99792700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f99792700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8f94136b60 con 0x7f8f94098920 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f7affd700 1 -- 172.21.15.6:0/3812283120 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8f8401c360 con 0x7f8f94098920 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f7affd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:13.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.714+0000 7f8f7affd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:13.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:13.716 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 >> v2:172.21.15.22:6812/142603 conn(0x7f8f800357a0 msgr2=0x7f8f80037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:13.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 --2- 172.21.15.6:0/3812283120 >> v2:172.21.15.22:6812/142603 conn(0x7f8f800357a0 0x7f8f80037c50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f8f940a1b40 tx=0x7f8f88016000 comp rx=0 tx=0).stop 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 10 monclient: shutdown 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 msgr2=0x7f8f9412dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 0x7f8f9412dd80 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8f8400d890 tx=0x7f8f84008040 comp rx=0 tx=0).stop 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 shutdown_connections 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 --2- 172.21.15.6:0/3812283120 >> v2:172.21.15.22:6812/142603 conn(0x7f8f800357a0 0x7f8f80037c50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8f94098dc0 0x7f8f9412fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8f94098550 0x7f8f9409e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 --2- 172.21.15.6:0/3812283120 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8f94098920 0x7f8f9412dd80 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:13.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.716+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 >> 172.21.15.6:0/3812283120 conn(0x7f8f940a5480 msgr2=0x7f8f94069120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.717+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 shutdown_connections 2023-09-30T00:47:13.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:13.717+0000 7f8f99792700 1 -- 172.21.15.6:0/3812283120 wait complete. 2023-09-30T00:47:16.732 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.869+0000 7fd038a64700 10 monclient: get_monmap_and_config 2023-09-30T00:47:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.869+0000 7fd038a64700 10 monclient: build_initial_monmap 2023-09-30T00:47:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.869+0000 7fd038a64700 10 monclient: monmap: 2023-09-30T00:47:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:16.870 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:16.870802+0000 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:16.870802+0000 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:16.871 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 1 Processor -- start 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 1 -- start start 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 10 monclient: init 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 0x7fd034132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.870+0000 7fd038a64700 10 monclient(hunting): picked mon.noname-b con 0x7fd034098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd034098a40 0x7fd03412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): picked mon.noname-a con 0x7fd034098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:16.876 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fd034098e90 0x7fd03412f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): picked mon.noname-c con 0x7fd034098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0340a0a50 con 0x7fd034098a40 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd0340a1000 con 0x7fd034098e90 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340a0730 con 0x7fd034098670 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd038a64700 10 monclient(hunting): authenticate will time out at 1433.169076s 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd034098a40 0x7fd03412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 0x7fd034132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.877 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd034098a40 0x7fd03412ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34688/0 (socket says 172.21.15.6:34688) 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 0x7fd034132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:39842/0 (socket says 172.21.15.6:39842) 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 -- 172.21.15.6:0/2612222661 learned_addr learned my addr 172.21.15.6:0/2612222661 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): get_auth_request con 0x7fd034098a40 auth_method 0 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): get_auth_request con 0x7fd034098670 auth_method 0 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:16.878 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd032d9d700 1 --2- 172.21.15.6:0/2612222661 >> v2:172.21.15.22:3300/0 conn(0x7fd034098e90 0x7fd03412f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd032d9d700 10 monclient(hunting): get_auth_request con 0x7fd034098e90 auth_method 0 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd032d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd032d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd032d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient(hunting): handle_auth_done global_id 4548 payload 274 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 -- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 msgr2=0x7fd034132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:16.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 --2- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 0x7fd034132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 -- 172.21.15.6:0/2612222661 >> v2:172.21.15.22:3300/0 conn(0x7fd034098e90 msgr2=0x7fd03412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 --2- 172.21.15.6:0/2612222661 >> v2:172.21.15.22:3300/0 conn(0x7fd034098e90 0x7fd03412f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 monclient: found mon.noname-a 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd031d9b700 1 -- 172.21.15.6:0/2612222661 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd034068df0 con 0x7fd034098a40 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.871+0000 7fd03259c700 1 --2- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 0x7fd034132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.873+0000 7fd031d9b700 1 -- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 msgr2=0x7fd03412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.873+0000 7fd031d9b700 1 --2- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412ede0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0340a1df0 tx=0x7fd028009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bcb25b2fd05ea17 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 1 -- 172.21.15.6:0/2612222661 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd028009e70 con 0x7fd034098a40 2023-09-30T00:47:16.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: dump: 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:16.881 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: _finish_auth 0 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 5 monclient: authenticate success, global_id 4548 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 1 -- 172.21.15.6:0/2612222661 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02800ba70 con 0x7fd034098a40 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 4 set_mon_vals no callback set 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 1 -- 172.21.15.6:0/2612222661 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02800be30 con 0x7fd034098a40 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:16.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd03159a700 10 monclient: dump: 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 10 monclient: shutdown 2023-09-30T00:47:16.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 -- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 msgr2=0x7fd03412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 --2- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412ede0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fd0340a1df0 tx=0x7fd028009250 comp rx=0 tx=0).stop 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 -- 172.21.15.6:0/2612222661 shutdown_connections 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 --2- 172.21.15.6:0/2612222661 >> v2:172.21.15.22:3300/0 conn(0x7fd034098e90 0x7fd03412f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 --2- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098670 0x7fd034132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 --2- 172.21.15.6:0/2612222661 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412ede0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 -- 172.21.15.6:0/2612222661 >> 172.21.15.6:0/2612222661 conn(0x7fd0340a5480 msgr2=0x7fd0340a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 -- 172.21.15.6:0/2612222661 shutdown_connections 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.874+0000 7fd038a64700 1 -- 172.21.15.6:0/2612222661 wait complete. 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.875+0000 7fd038a64700 10 monclient: build_initial_monmap 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.875+0000 7fd038a64700 10 monclient: monmap: 2023-09-30T00:47:16.884 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:16.885 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 Processor -- start 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 -- start start 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient: init 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd034098670 0x7fd03412d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): picked mon.noname-b con 0x7fd034098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): picked mon.noname-a con 0x7fd034098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 0x7fd03409d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): picked mon.noname-c con 0x7fd034098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd03409d290 con 0x7fd034098a40 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0340a1500 con 0x7fd034098670 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd034135390 con 0x7fd034098e90 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.876+0000 7fd038a64700 10 monclient(hunting): authenticate will time out at 1433.174929s 2023-09-30T00:47:16.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 0x7fd03409d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34702/0 (socket says 172.21.15.6:34702) 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 1 -- 172.21.15.6:0/211341424 learned_addr learned my addr 172.21.15.6:0/211341424 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): get_auth_request con 0x7fd034098a40 auth_method 0 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): get_auth_request con 0x7fd034098e90 auth_method 0 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:16.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd034098670 0x7fd03412d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): get_auth_request con 0x7fd034098670 auth_method 0 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:16.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd03259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient(hunting): handle_auth_done global_id 4556 payload 824 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 -- 172.21.15.6:0/211341424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd034098670 msgr2=0x7fd03412d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd034098670 0x7fd03412d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 -- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 msgr2=0x7fd03412d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 monclient: found mon.noname-c 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd032d9d700 1 -- 172.21.15.6:0/211341424 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd024009420 con 0x7fd034098e90 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.877+0000 7fd031d9b700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd032d9d700 1 -- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 msgr2=0x7fd03409d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd032d9d700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 0x7fd03409d920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd02c007ed0 tx=0x7fd02c009850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c00eda0 con 0x7fd034098e90 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: dump: 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 5 monclient: authenticate success, global_id 4556 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd02c014420 con 0x7fd034098e90 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 10 monclient: _renew_subs 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0280094f0 con 0x7fd034098e90 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd02c0147e0 con 0x7fd034098e90 2023-09-30T00:47:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd030d99700 4 set_mon_vals no callback set 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd0227fc700 10 monclient: dump: 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:16.897 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 10 monclient: _renew_subs 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.879+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd03412e730 con 0x7fd034098e90 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.880+0000 7fd038a64700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.880+0000 7fd038a64700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.880+0000 7fd038a64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.880+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd02c003020 con 0x7fd034098e90 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.880+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd03409d0f0 con 0x7fd034098e90 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.881+0000 7fd0227fc700 1 --2- 172.21.15.6:0/211341424 >> v2:172.21.15.22:6812/142603 conn(0x7fd0140357a0 0x7fd014037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.881+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd02c047c90 con 0x7fd034098e90 2023-09-30T00:47:16.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.881+0000 7fd03259c700 1 --2- 172.21.15.6:0/211341424 >> v2:172.21.15.22:6812/142603 conn(0x7fd0140357a0 0x7fd014037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:16.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.881+0000 7fd03259c700 10 monclient: get_auth_request con 0x7fd0140357a0 auth_method 0 2023-09-30T00:47:16.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.884+0000 7fd03259c700 1 --2- 172.21.15.6:0/211341424 >> v2:172.21.15.22:6812/142603 conn(0x7fd0140357a0 0x7fd014037c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd024009f40 tx=0x7fd0240091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:16.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.886+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd03409d0f0 con 0x7fd034098e90 2023-09-30T00:47:16.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.886+0000 7fd0227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:16.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:16.886+0000 7fd0227fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:17.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.134+0000 7fd038a64700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:17.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.134+0000 7fd038a64700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:17.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.134+0000 7fd038a64700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:17.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.134+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd03409d0f0 con 0x7fd034098e90 2023-09-30T00:47:17.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.135+0000 7fd0227fc700 1 -- 172.21.15.6:0/211341424 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd03409d0f0 con 0x7fd034098e90 2023-09-30T00:47:17.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.135+0000 7fd0227fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:17.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.135+0000 7fd0227fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:17.136 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:17.136 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:17.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 >> v2:172.21.15.22:6812/142603 conn(0x7fd0140357a0 msgr2=0x7fd014037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:17.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 --2- 172.21.15.6:0/211341424 >> v2:172.21.15.22:6812/142603 conn(0x7fd0140357a0 0x7fd014037c50 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fd024009f40 tx=0x7fd0240091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:17.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 10 monclient: shutdown 2023-09-30T00:47:17.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 msgr2=0x7fd03409d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 0x7fd03409d920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fd02c007ed0 tx=0x7fd02c009850 comp rx=0 tx=0).stop 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 shutdown_connections 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 --2- 172.21.15.6:0/211341424 >> v2:172.21.15.22:6812/142603 conn(0x7fd0140357a0 0x7fd014037c50 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd034098e90 0x7fd03409d920 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd034098670 0x7fd03412d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 --2- 172.21.15.6:0/211341424 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd034098a40 0x7fd03412d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 >> 172.21.15.6:0/211341424 conn(0x7fd0340a5480 msgr2=0x7fd034069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 shutdown_connections 2023-09-30T00:47:17.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:17.137+0000 7fd038a64700 1 -- 172.21.15.6:0/211341424 wait complete. 2023-09-30T00:47:20.152 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.292+0000 7fce3c932700 10 monclient: get_monmap_and_config 2023-09-30T00:47:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.292+0000 7fce3c932700 10 monclient: build_initial_monmap 2023-09-30T00:47:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.292+0000 7fce3c932700 10 monclient: monmap: 2023-09-30T00:47:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:20.293 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:20.293834+0000 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:20.293834+0000 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:20.294 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 1 Processor -- start 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 1 -- start start 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 10 monclient: init 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fce38098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fce38132d00 0x7fce38137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:20.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.293+0000 7fce3c932700 10 monclient(hunting): picked mon.noname-c con 0x7fce38132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce38137b60 0x7fce38139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 10 monclient(hunting): picked mon.noname-a con 0x7fce38137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce380a1000 con 0x7fce38098670 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce38099150 con 0x7fce38137b60 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce380a0a50 con 0x7fce38132d00 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce3c932700 10 monclient(hunting): authenticate will time out at 1436.592046s 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce38137b60 0x7fce38139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:20.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fce38137b60 0x7fce38139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34716/0 (socket says 172.21.15.6:34716) 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 -- 172.21.15.6:0/1232516132 learned_addr learned my addr 172.21.15.6:0/1232516132 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): get_auth_request con 0x7fce38137b60 auth_method 0 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce35d9b700 1 --2- 172.21.15.6:0/1232516132 >> v2:172.21.15.22:3300/0 conn(0x7fce38132d00 0x7fce38137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce35d9b700 10 monclient(hunting): get_auth_request con 0x7fce38132d00 auth_method 0 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce35d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:20.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce35d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce35d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient(hunting): handle_auth_done global_id 4554 payload 274 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 -- 172.21.15.6:0/1232516132 >> v2:172.21.15.22:3300/0 conn(0x7fce38132d00 msgr2=0x7fce38137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 --2- 172.21.15.6:0/1232516132 >> v2:172.21.15.22:3300/0 conn(0x7fce38132d00 0x7fce38137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 -- 172.21.15.6:0/1232516132 >> v1:172.21.15.22:6789/0 conn(0x7fce38098670 legacy=0x7fce38098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 monclient: found mon.noname-a 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce36d9d700 1 -- 172.21.15.6:0/1232516132 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce380a0730 con 0x7fce38137b60 2023-09-30T00:47:20.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.294+0000 7fce35d9b700 1 --2- 172.21.15.6:0/1232516132 >> v2:172.21.15.22:3300/0 conn(0x7fce38132d00 0x7fce38137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce36d9d700 1 -- 172.21.15.6:0/1232516132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 msgr2=0x7fce38139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce36d9d700 1 --2- 172.21.15.6:0/1232516132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce38139f40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce38063790 tx=0x7fce30009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb74451d4f17de0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 1 -- 172.21.15.6:0/1232516132 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce3000e070 con 0x7fce38137b60 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: dump: 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.302 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: _finish_auth 0 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3c932700 5 monclient: authenticate success, global_id 4554 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 1 -- 172.21.15.6:0/1232516132 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce3000bc20 con 0x7fce38137b60 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:20.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3c932700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3c932700 4 set_mon_vals no callback set 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 1 -- 172.21.15.6:0/1232516132 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce300092e0 con 0x7fce38137b60 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.296+0000 7fce3559a700 10 monclient: dump: 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:20.304 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 10 monclient: shutdown 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 -- 172.21.15.6:0/1232516132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 msgr2=0x7fce38139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 --2- 172.21.15.6:0/1232516132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce38139f40 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce38063790 tx=0x7fce30009840 comp rx=0 tx=0).stop 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 -- 172.21.15.6:0/1232516132 shutdown_connections 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 --2- 172.21.15.6:0/1232516132 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce38139f40 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 --2- 172.21.15.6:0/1232516132 >> v2:172.21.15.22:3300/0 conn(0x7fce38132d00 0x7fce38137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 -- 172.21.15.6:0/1232516132 >> 172.21.15.6:0/1232516132 conn(0x7fce380a5480 msgr2=0x7fce380a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:20.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 -- 172.21.15.6:0/1232516132 shutdown_connections 2023-09-30T00:47:20.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.297+0000 7fce3c932700 1 -- 172.21.15.6:0/1232516132 wait complete. 2023-09-30T00:47:20.310 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.298+0000 7fce3c932700 10 monclient: build_initial_monmap 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.298+0000 7fce3c932700 10 monclient: monmap: 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:20.311 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.298+0000 7fce3c932700 1 Processor -- start 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 -- start start 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient: init 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 0x7fce3809bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): picked mon.noname-c con 0x7fce38098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:20.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce38132d00 0x7fce38133ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): picked mon.noname-b con 0x7fce38132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce381340e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): picked mon.noname-a con 0x7fce38137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce38136ed0 con 0x7fce38137b60 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce380a1500 con 0x7fce38132d00 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce381300a0 con 0x7fce38098670 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3c932700 10 monclient(hunting): authenticate will time out at 1436.597775s 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 0x7fce3809bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 0x7fce3809bb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:39868/0 (socket says 172.21.15.6:39868) 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 1 -- 172.21.15.6:0/2689528905 learned_addr learned my addr 172.21.15.6:0/2689528905 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): get_auth_request con 0x7fce38098670 auth_method 0 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce36d9d700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce381340e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce35d9b700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce38132d00 0x7fce38133ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce36d9d700 10 monclient(hunting): get_auth_request con 0x7fce38137b60 auth_method 0 2023-09-30T00:47:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce36d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce36d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce36d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.299+0000 7fce3659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): get_auth_request con 0x7fce38132d00 auth_method 0 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce36d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce36d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:20.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce36d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce35d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 10 monclient(hunting): handle_auth_done global_id 4559 payload 824 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 1 -- 172.21.15.6:0/2689528905 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce38132d00 msgr2=0x7fce38133ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce38132d00 0x7fce38133ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 1 -- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 msgr2=0x7fce381340e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce381340e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 10 monclient: _finish_hunting 0 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 1 monclient: found mon.noname-c 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce3659c700 1 -- 172.21.15.6:0/2689528905 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce30009ae0 con 0x7fce38098670 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.300+0000 7fce36d9d700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce381340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3659c700 1 -- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 msgr2=0x7fce3809bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3659c700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 0x7fce3809bb60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fce2800c5d0 tx=0x7fce28007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce280117a0 con 0x7fce38098670 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: dump: 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce28011900 con 0x7fce38098670 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce28011cc0 con 0x7fce38098670 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:20.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce267fc700 10 monclient: dump: 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:20.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 5 monclient: authenticate success, global_id 4559 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 10 monclient: _renew_subs 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce34d99700 4 set_mon_vals no callback set 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce3812fc30 con 0x7fce38098670 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 10 monclient: _renew_subs 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.302+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce3812ed40 con 0x7fce38098670 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fce28012630 con 0x7fce38098670 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce3c932700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce3c932700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce3c932700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:20.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce3812e6e0 con 0x7fce38098670 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce267fc700 1 --2- 172.21.15.6:0/2689528905 >> v2:172.21.15.22:6812/142603 conn(0x7fce180357a0 0x7fce18037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.303+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fce28047680 con 0x7fce38098670 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.304+0000 7fce35d9b700 1 --2- 172.21.15.6:0/2689528905 >> v2:172.21.15.22:6812/142603 conn(0x7fce180357a0 0x7fce18037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.304+0000 7fce35d9b700 10 monclient: get_auth_request con 0x7fce180357a0 auth_method 0 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.306+0000 7fce35d9b700 1 --2- 172.21.15.6:0/2689528905 >> v2:172.21.15.22:6812/142603 conn(0x7fce180357a0 0x7fce18037c50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce2c006f00 tx=0x7fce2c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.308+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fce3812e6e0 con 0x7fce38098670 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.308+0000 7fce267fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:20.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.308+0000 7fce267fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.551+0000 7fce3c932700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.551+0000 7fce3c932700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:20.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.551+0000 7fce3c932700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:20.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.551+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fce3812efc0 con 0x7fce38098670 2023-09-30T00:47:20.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.552+0000 7fce267fc700 1 -- 172.21.15.6:0/2689528905 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fce280108d0 con 0x7fce38098670 2023-09-30T00:47:20.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.552+0000 7fce267fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:20.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.552+0000 7fce267fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:20.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:20.553 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 >> v2:172.21.15.22:6812/142603 conn(0x7fce180357a0 msgr2=0x7fce18037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 --2- 172.21.15.6:0/2689528905 >> v2:172.21.15.22:6812/142603 conn(0x7fce180357a0 0x7fce18037c50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fce2c006f00 tx=0x7fce2c008040 comp rx=0 tx=0).stop 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 10 monclient: shutdown 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 msgr2=0x7fce3809bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 0x7fce3809bb60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fce2800c5d0 tx=0x7fce28007f60 comp rx=0 tx=0).stop 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 shutdown_connections 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 --2- 172.21.15.6:0/2689528905 >> v2:172.21.15.22:6812/142603 conn(0x7fce180357a0 0x7fce18037c50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fce38137b60 0x7fce381340e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fce38098670 0x7fce3809bb60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 --2- 172.21.15.6:0/2689528905 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fce38132d00 0x7fce38133ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:20.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 >> 172.21.15.6:0/2689528905 conn(0x7fce380a5480 msgr2=0x7fce38130660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:20.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 shutdown_connections 2023-09-30T00:47:20.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:20.554+0000 7fce3c932700 1 -- 172.21.15.6:0/2689528905 wait complete. 2023-09-30T00:47:23.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:23.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.705+0000 7fd612d73700 10 monclient: get_monmap_and_config 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.705+0000 7fd612d73700 10 monclient: build_initial_monmap 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.705+0000 7fd612d73700 10 monclient: monmap: 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:23.706355+0000 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:23.706355+0000 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:23.706 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:23.707 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:23.707 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.705+0000 7fd612d73700 1 Processor -- start 2023-09-30T00:47:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 1 -- start start 2023-09-30T00:47:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient: init 2023-09-30T00:47:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:47:23.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd60c098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd60c132d00 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fd60c138d40 0x7fd60c13b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient(hunting): picked mon.noname-c con 0x7fd60c138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0a1000 con 0x7fd60c132d00 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd60c0a0730 con 0x7fd60c098670 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd60c099150 con 0x7fd60c138d40 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd612d73700 10 monclient(hunting): authenticate will time out at 1440.004580s 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd60bfff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd60c132d00 0x7fd60c137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41080/0 (socket says 172.21.15.6:41080) 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd60bfff700 1 -- 172.21.15.6:0/3150345529 learned_addr learned my addr 172.21.15.6:0/3150345529 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:23.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd611310700 1 --2- 172.21.15.6:0/3150345529 >> v2:172.21.15.22:3300/0 conn(0x7fd60c138d40 0x7fd60c13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd611310700 10 monclient(hunting): get_auth_request con 0x7fd60c138d40 auth_method 0 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd611310700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd611310700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.706+0000 7fd611310700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 10 monclient(hunting): handle_auth_done global_id 4558 payload 274 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 1 -- 172.21.15.6:0/3150345529 >> v1:172.21.15.22:6789/0 conn(0x7fd60c098670 legacy=0x7fd60c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 1 -- 172.21.15.6:0/3150345529 >> v1:172.21.15.6:6789/0 conn(0x7fd60c132d00 legacy=0x7fd60c137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:23.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 10 monclient: _finish_hunting 0 2023-09-30T00:47:23.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 1 monclient: found mon.noname-c 2023-09-30T00:47:23.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:23.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:47:23.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.707+0000 7fd611310700 1 -- 172.21.15.6:0/3150345529 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd60c068df0 con 0x7fd60c138d40 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.709+0000 7fd611310700 1 -- 172.21.15.6:0/3150345529 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c138d40 msgr2=0x7fd60c13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.709+0000 7fd611310700 1 --2- 172.21.15.6:0/3150345529 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c138d40 0x7fd60c13b120 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd604009e60 tx=0x7fd604009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1bdbd0c9897d88 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 1 -- 172.21.15.6:0/3150345529 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd60400e070 con 0x7fd60c138d40 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: dump: 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:23.718 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:23.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 5 monclient: authenticate success, global_id 4558 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 1 -- 172.21.15.6:0/3150345529 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd604012430 con 0x7fd60c138d40 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 4 set_mon_vals no callback set 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 1 -- 172.21.15.6:0/3150345529 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6040127d0 con 0x7fd60c138d40 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd60b7fe700 10 monclient: dump: 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:23.720 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 10 monclient: shutdown 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 -- 172.21.15.6:0/3150345529 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c138d40 msgr2=0x7fd60c13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 --2- 172.21.15.6:0/3150345529 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c138d40 0x7fd60c13b120 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd604009e60 tx=0x7fd604009780 comp rx=0 tx=0).stop 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 -- 172.21.15.6:0/3150345529 shutdown_connections 2023-09-30T00:47:23.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 --2- 172.21.15.6:0/3150345529 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c138d40 0x7fd60c13b120 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 -- 172.21.15.6:0/3150345529 >> 172.21.15.6:0/3150345529 conn(0x7fd60c0a5480 msgr2=0x7fd60c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 -- 172.21.15.6:0/3150345529 shutdown_connections 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.710+0000 7fd612d73700 1 -- 172.21.15.6:0/3150345529 wait complete. 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.711+0000 7fd612d73700 10 monclient: build_initial_monmap 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.711+0000 7fd612d73700 10 monclient: monmap: 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:23.722 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 1 Processor -- start 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 1 -- start start 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient: init 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 0x7fd60c141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient(hunting): picked mon.noname-a con 0x7fd60c098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:23.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c132d00 0x7fd60c138310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient(hunting): picked mon.noname-b con 0x7fd60c132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd60c138d40 0x7fd60c13f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient(hunting): picked mon.noname-c con 0x7fd60c138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.712+0000 7fd612d73700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c130400 con 0x7fd60c098670 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd612d73700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd612d73700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c133ad0 con 0x7fd60c132d00 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd612d73700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd612d73700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd60c12f5a0 con 0x7fd60c138d40 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd612d73700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd612d73700 10 monclient(hunting): authenticate will time out at 1440.011048s 2023-09-30T00:47:23.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 0x7fd60c141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 0x7fd60c141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:34726/0 (socket says 172.21.15.6:34726) 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 -- 172.21.15.6:0/855693910 learned_addr learned my addr 172.21.15.6:0/855693910 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): get_auth_request con 0x7fd60c098670 auth_method 0 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:23.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd60c138d40 0x7fd60c13f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): get_auth_request con 0x7fd60c138d40 auth_method 0 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c132d00 0x7fd60c138310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): get_auth_request con 0x7fd60c132d00 auth_method 0 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:23.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd60bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient(hunting): handle_auth_done global_id 4560 payload 824 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 -- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd60c138d40 msgr2=0x7fd60c13f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd60c138d40 0x7fd60c13f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 -- 172.21.15.6:0/855693910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c132d00 msgr2=0x7fd60c138310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:23.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c132d00 0x7fd60c138310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient: _finish_hunting 0 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 monclient: found mon.noname-a 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd610b0f700 1 -- 172.21.15.6:0/855693910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd604009a20 con 0x7fd60c098670 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.713+0000 7fd611310700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd60c138d40 0x7fd60c13f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.715+0000 7fd610b0f700 1 -- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 msgr2=0x7fd60c141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.715+0000 7fd610b0f700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 0x7fd60c141600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c5d0 tx=0x7fd5fc007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.715+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc0117a0 con 0x7fd60c098670 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.715+0000 7fd608ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.715+0000 7fd608ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:23.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.715+0000 7fd608ff9700 10 monclient: dump: 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:23.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 5 monclient: authenticate success, global_id 4560 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd5fc011920 con 0x7fd60c098670 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 monclient: _renew_subs 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd60c133510 con 0x7fd60c098670 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5fc011ce0 con 0x7fd60c098670 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 10 monclient: dump: 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:23.730 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd60affd700 4 set_mon_vals no callback set 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 monclient: _renew_subs 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd60c12e8e0 con 0x7fd60c098670 2023-09-30T00:47:23.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd5fc012630 con 0x7fd60c098670 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.716+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd60c072b00 con 0x7fd60c098670 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.717+0000 7fd608ff9700 1 --2- 172.21.15.6:0/855693910 >> v2:172.21.15.22:6812/142603 conn(0x7fd5f80357a0 0x7fd5f8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.717+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd5fc047660 con 0x7fd60c098670 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.717+0000 7fd60bfff700 1 --2- 172.21.15.6:0/855693910 >> v2:172.21.15.22:6812/142603 conn(0x7fd5f80357a0 0x7fd5f8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:23.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.717+0000 7fd60bfff700 10 monclient: get_auth_request con 0x7fd5f80357a0 auth_method 0 2023-09-30T00:47:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.720+0000 7fd60bfff700 1 --2- 172.21.15.6:0/855693910 >> v2:172.21.15.22:6812/142603 conn(0x7fd5f80357a0 0x7fd5f8037c50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0a5350 tx=0x7fd600007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.721+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd5fc011e40 con 0x7fd60c098670 2023-09-30T00:47:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.721+0000 7fd608ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.721+0000 7fd608ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd612d73700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd612d73700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd612d73700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:23.968 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd60c063920 con 0x7fd60c098670 2023-09-30T00:47:23.969 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd608ff9700 1 -- 172.21.15.6:0/855693910 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd5fc0102e0 con 0x7fd60c098670 2023-09-30T00:47:23.969 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd608ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:23.969 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.967+0000 7fd608ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:23.969 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:23.969 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:23.970 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 >> v2:172.21.15.22:6812/142603 conn(0x7fd5f80357a0 msgr2=0x7fd5f8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 1 --2- 172.21.15.6:0/855693910 >> v2:172.21.15.22:6812/142603 conn(0x7fd5f80357a0 0x7fd5f8037c50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fd60c0a5350 tx=0x7fd600007f90 comp rx=0 tx=0).stop 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 10 monclient: shutdown 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 msgr2=0x7fd60c141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 0x7fd60c141600 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fd5fc00c5d0 tx=0x7fd5fc007f60 comp rx=0 tx=0).stop 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 shutdown_connections 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.969+0000 7fd612d73700 1 --2- 172.21.15.6:0/855693910 >> v2:172.21.15.22:6812/142603 conn(0x7fd5f80357a0 0x7fd5f8037c50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.970+0000 7fd612d73700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd60c138d40 0x7fd60c13f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.970+0000 7fd612d73700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd60c098670 0x7fd60c141600 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.970+0000 7fd612d73700 1 --2- 172.21.15.6:0/855693910 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd60c132d00 0x7fd60c138310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:23.971 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.970+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 >> 172.21.15.6:0/855693910 conn(0x7fd60c0a5480 msgr2=0x7fd60c1305a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:23.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.970+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 shutdown_connections 2023-09-30T00:47:23.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:23.970+0000 7fd612d73700 1 -- 172.21.15.6:0/855693910 wait complete. 2023-09-30T00:47:26.984 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.121+0000 7f8cd494c700 10 monclient: get_monmap_and_config 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.121+0000 7f8cd494c700 10 monclient: build_initial_monmap 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.121+0000 7f8cd494c700 10 monclient: monmap: 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:27.122242+0000 2023-09-30T00:47:27.122 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:27.122242+0000 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.121+0000 7f8cd494c700 1 Processor -- start 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.121+0000 7f8cd494c700 1 -- start start 2023-09-30T00:47:27.123 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.121+0000 7f8cd494c700 10 monclient: init 2023-09-30T00:47:27.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:27.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient: _add_conns ranks=[3,2,4,0,1] 2023-09-30T00:47:27.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8cd0098670 0x7f8cd0132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): picked mon.noname-a con 0x7f8cd0098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:27.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): picked mon.noname-b con 0x7f8cd0098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8cd0098e90 0x7f8cd012f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): picked mon.noname-c con 0x7f8cd0098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00a0a50 con 0x7f8cd0098670 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd00a1000 con 0x7f8cd0098e90 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00a0730 con 0x7f8cd0098a40 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cd494c700 10 monclient(hunting): authenticate will time out at 1443.420370s 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8cd0098670 0x7f8cd0132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8cd0098670 0x7f8cd0132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39416/0 (socket says 172.21.15.6:39416) 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34046/0 (socket says 172.21.15.6:34046) 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 -- 172.21.15.6:0/3386519576 learned_addr learned my addr 172.21.15.6:0/3386519576 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): get_auth_request con 0x7f8cd0098670 auth_method 0 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): get_auth_request con 0x7f8cd0098a40 auth_method 0 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:27.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cced9d700 1 --2- 172.21.15.6:0/3386519576 >> v2:172.21.15.22:3300/0 conn(0x7f8cd0098e90 0x7f8cd012f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient(hunting): handle_auth_done global_id 4563 payload 274 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 -- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 msgr2=0x7f8cd012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 --2- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 -- 172.21.15.6:0/3386519576 >> v2:172.21.15.22:3300/0 conn(0x7f8cd0098e90 msgr2=0x7f8cd012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 --2- 172.21.15.6:0/3386519576 >> v2:172.21.15.22:3300/0 conn(0x7f8cd0098e90 0x7f8cd012f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient: _finish_hunting 0 2023-09-30T00:47:27.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 monclient: found mon.noname-a 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cce59c700 1 -- 172.21.15.6:0/3386519576 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd0068df0 con 0x7f8cd0098670 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cced9d700 10 monclient: get_auth_request con 0x7f8cd0098e90 auth_method 0 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8cced9d700 1 --2- 172.21.15.6:0/3386519576 >> v2:172.21.15.22:3300/0 conn(0x7f8cd0098e90 0x7f8cd012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.122+0000 7f8ccdd9b700 1 --2- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cce59c700 1 -- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098670 msgr2=0x7f8cd0132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cce59c700 1 --2- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098670 0x7f8cd0132d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00a1df0 tx=0x7f8cc0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5a014aacadcf675 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 1 -- 172.21.15.6:0/3386519576 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc0009e70 con 0x7f8cd0098670 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:27.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: dump: 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: _finish_auth 0 2023-09-30T00:47:27.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 5 monclient: authenticate success, global_id 4563 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 1 -- 172.21.15.6:0/3386519576 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc000ba70 con 0x7f8cd0098670 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 4 set_mon_vals no callback set 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 1 -- 172.21.15.6:0/3386519576 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc000be30 con 0x7f8cd0098670 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8ccd59a700 10 monclient: dump: 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:27.133 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 10 monclient: shutdown 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 1 -- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098670 msgr2=0x7f8cd0132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.125+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098670 0x7f8cd0132d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00a1df0 tx=0x7f8cc0009250 comp rx=0 tx=0).stop 2023-09-30T00:47:27.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 -- 172.21.15.6:0/3386519576 shutdown_connections 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3386519576 >> v2:172.21.15.22:3300/0 conn(0x7f8cd0098e90 0x7f8cd012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098670 0x7f8cd0132d00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3386519576 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 -- 172.21.15.6:0/3386519576 >> 172.21.15.6:0/3386519576 conn(0x7f8cd00a5480 msgr2=0x7f8cd00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 -- 172.21.15.6:0/3386519576 shutdown_connections 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 -- 172.21.15.6:0/3386519576 wait complete. 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 10 monclient: build_initial_monmap 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 10 monclient: monmap: 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:27.135 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.126+0000 7f8cd494c700 1 Processor -- start 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 -- start start 2023-09-30T00:47:27.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient: init 2023-09-30T00:47:27.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:27.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cd0098670 0x7f8cd012d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): picked mon.noname-b con 0x7f8cd0098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): picked mon.noname-c con 0x7f8cd0098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 0x7f8cd009d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): picked mon.noname-a con 0x7f8cd0098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd009ccf0 con 0x7f8cd0098e90 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd009d220 con 0x7f8cd0098670 2023-09-30T00:47:27.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd00a1500 con 0x7f8cd0098a40 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cd494c700 10 monclient(hunting): authenticate will time out at 1443.425565s 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 0x7f8cd009d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 0x7f8cd009d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39432/0 (socket says 172.21.15.6:39432) 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34048/0 (socket says 172.21.15.6:34048) 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 1 -- 172.21.15.6:0/3886162399 learned_addr learned my addr 172.21.15.6:0/3886162399 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): get_auth_request con 0x7f8cd0098e90 auth_method 0 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:27.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): get_auth_request con 0x7f8cd0098a40 auth_method 0 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cd0098670 0x7f8cd012d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): get_auth_request con 0x7f8cd0098670 auth_method 0 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:27.143 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cced9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.127+0000 7f8cce59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 10 monclient(hunting): handle_auth_done global_id 4568 payload 824 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 1 -- 172.21.15.6:0/3886162399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cd0098670 msgr2=0x7f8cd012d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cd0098670 0x7f8cd012d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 1 -- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 msgr2=0x7f8cd009d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 0x7f8cd009d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:27.144 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 1 monclient: found mon.noname-c 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8ccdd9b700 1 -- 172.21.15.6:0/3886162399 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc4009420 con 0x7f8cd0098a40 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.128+0000 7f8cced9d700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 0x7f8cd009d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8ccdd9b700 1 -- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 msgr2=0x7f8cd012d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8ccdd9b700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400ecb0 tx=0x7f8cc40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc4009700 con 0x7f8cd0098a40 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: dump: 2023-09-30T00:47:27.145 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:27.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 5 monclient: authenticate success, global_id 4568 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cc400ca60 con 0x7f8cd0098a40 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 10 monclient: _renew_subs 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc00094f0 con 0x7f8cd0098a40 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8cc400ce00 con 0x7f8cd0098a40 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cbe7fc700 10 monclient: dump: 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.147 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cccd99700 4 set_mon_vals no callback set 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 10 monclient: _renew_subs 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.130+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd012e6e0 con 0x7f8cd0098a40 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8ccdd9b700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400ecb0 tx=0x7f8cc40091a0 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f8cc4013680 mgrmap(e 5) v1 2023-09-30T00:47:27.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8cc4013680 con 0x7f8cd0098a40 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8cd494c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8cd494c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8cd494c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd009d060 con 0x7f8cd0098a40 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.131+0000 7f8cbe7fc700 1 --2- 172.21.15.6:0/3886162399 >> v2:172.21.15.22:6812/142603 conn(0x7f8cb00357a0 0x7f8cb0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.132+0000 7f8cce59c700 1 --2- 172.21.15.6:0/3886162399 >> v2:172.21.15.22:6812/142603 conn(0x7f8cb00357a0 0x7f8cb0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.132+0000 7f8cce59c700 10 monclient: get_auth_request con 0x7f8cb00357a0 auth_method 0 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.134+0000 7f8cce59c700 1 --2- 172.21.15.6:0/3886162399 >> v2:172.21.15.22:6812/142603 conn(0x7f8cb00357a0 0x7f8cb0037c50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00a1df0 tx=0x7f8cc0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:27.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.136+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8cc4047c50 con 0x7f8cd0098a40 2023-09-30T00:47:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.960+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8cc40123d0 con 0x7f8cd0098a40 2023-09-30T00:47:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.960+0000 7f8cbe7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:27.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:27.960+0000 7f8cbe7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cd494c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cd494c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cd494c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8cd0072b00 con 0x7f8cd0098a40 2023-09-30T00:47:28.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cbe7fc700 1 -- 172.21.15.6:0/3886162399 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8cc401a360 con 0x7f8cd0098a40 2023-09-30T00:47:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cbe7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:28.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.213+0000 7f8cbe7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:28.215 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:28.215 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 >> v2:172.21.15.22:6812/142603 conn(0x7f8cb00357a0 msgr2=0x7f8cb0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3886162399 >> v2:172.21.15.22:6812/142603 conn(0x7f8cb00357a0 0x7f8cb0037c50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f8cd00a1df0 tx=0x7f8cc0017000 comp rx=0 tx=0).stop 2023-09-30T00:47:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 10 monclient: shutdown 2023-09-30T00:47:28.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 msgr2=0x7f8cd012d9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8cc400ecb0 tx=0x7f8cc40091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 shutdown_connections 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3886162399 >> v2:172.21.15.22:6812/142603 conn(0x7f8cb00357a0 0x7f8cb0037c50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8cd0098e90 0x7f8cd009d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8cd0098670 0x7f8cd012d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 --2- 172.21.15.6:0/3886162399 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8cd0098a40 0x7f8cd012d9d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 >> 172.21.15.6:0/3886162399 conn(0x7f8cd00a5480 msgr2=0x7f8cd0069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 shutdown_connections 2023-09-30T00:47:28.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:28.215+0000 7f8cd494c700 1 -- 172.21.15.6:0/3886162399 wait complete. 2023-09-30T00:47:31.230 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.369+0000 7fa4556b2700 10 monclient: get_monmap_and_config 2023-09-30T00:47:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.369+0000 7fa4556b2700 10 monclient: build_initial_monmap 2023-09-30T00:47:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.369+0000 7fa4556b2700 10 monclient: monmap: 2023-09-30T00:47:31.370 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:31.370629+0000 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:31.370629+0000 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:31.371 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:31.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 Processor -- start 2023-09-30T00:47:31.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 -- start start 2023-09-30T00:47:31.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient: init 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient: _add_conns ranks=[4,3,0,1,2] 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fa450132150 0x7fa450132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): picked mon.noname-c con 0x7fa450132150 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa450132e10 0x7fa450098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): picked mon.noname-a con 0x7fa450132e10 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa450098ca0 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4500990e0 con 0x7fa450098ca0 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa450132520 con 0x7fa450132e10 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:31.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa450096a80 con 0x7fa450132150 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa4556b2700 10 monclient(hunting): authenticate will time out at 1447.668678s 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa450132e10 0x7fa450098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fa450132e10 0x7fa450098760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39440/0 (socket says 172.21.15.6:39440) 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 1 -- 172.21.15.6:0/2024815226 learned_addr learned my addr 172.21.15.6:0/2024815226 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): get_auth_request con 0x7fa450132e10 auth_method 0 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44f7fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa450098ca0 0x7fa450068df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34988/0 (socket says 172.21.15.6:34988) 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:31.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.370+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 1 --2- 172.21.15.6:0/2024815226 >> v2:172.21.15.22:3300/0 conn(0x7fa450132150 0x7fa450132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): get_auth_request con 0x7fa450132150 auth_method 0 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient(hunting): handle_auth_done global_id 4567 payload 274 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 1 -- 172.21.15.6:0/2024815226 >> v2:172.21.15.6:3300/0 conn(0x7fa450132e10 msgr2=0x7fa450098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:31.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 1 --2- 172.21.15.6:0/2024815226 >> v2:172.21.15.6:3300/0 conn(0x7fa450132e10 0x7fa450098760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 1 -- 172.21.15.6:0/2024815226 >> v1:172.21.15.6:6789/0 conn(0x7fa450098ca0 legacy=0x7fa450068df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 1 monclient: found mon.noname-c 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.371+0000 7fa44effd700 1 -- 172.21.15.6:0/2024815226 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4500a0f80 con 0x7fa450132150 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44effd700 1 -- 172.21.15.6:0/2024815226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450132150 msgr2=0x7fa450132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44effd700 1 --2- 172.21.15.6:0/2024815226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450132150 0x7fa450132790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa45006a650 tx=0x7fa4400091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e263acb472381e7 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 1 -- 172.21.15.6:0/2024815226 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440011840 con 0x7fa450132150 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: dump: 2023-09-30T00:47:31.379 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:31.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 1 -- 172.21.15.6:0/2024815226 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4400119a0 con 0x7fa450132150 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa4556b2700 5 monclient: authenticate success, global_id 4567 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa4556b2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa4556b2700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 1 -- 172.21.15.6:0/2024815226 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa440011d40 con 0x7fa450132150 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa4556b2700 4 set_mon_vals no callback set 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.373+0000 7fa44dffb700 10 monclient: dump: 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:31.381 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 10 monclient: shutdown 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 -- 172.21.15.6:0/2024815226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450132150 msgr2=0x7fa450132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 --2- 172.21.15.6:0/2024815226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450132150 0x7fa450132790 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa45006a650 tx=0x7fa4400091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 -- 172.21.15.6:0/2024815226 shutdown_connections 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 --2- 172.21.15.6:0/2024815226 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450132150 0x7fa450132790 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 --2- 172.21.15.6:0/2024815226 >> v2:172.21.15.6:3300/0 conn(0x7fa450132e10 0x7fa450098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 -- 172.21.15.6:0/2024815226 >> 172.21.15.6:0/2024815226 conn(0x7fa4500a54c0 msgr2=0x7fa4500a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 -- 172.21.15.6:0/2024815226 shutdown_connections 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.374+0000 7fa4556b2700 1 -- 172.21.15.6:0/2024815226 wait complete. 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.375+0000 7fa4556b2700 10 monclient: build_initial_monmap 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.375+0000 7fa4556b2700 10 monclient: monmap: 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:31.383 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:31.384 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:31.384 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:31.384 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:31.384 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:31.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.375+0000 7fa4556b2700 1 Processor -- start 2023-09-30T00:47:31.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.375+0000 7fa4556b2700 1 -- start start 2023-09-30T00:47:31.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.375+0000 7fa4556b2700 10 monclient: init 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450098ca0 0x7fa45012fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): picked mon.noname-b con 0x7fa450098ca0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 0x7fa45011c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): picked mon.noname-a con 0x7fa450132150 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa450132e10 0x7fa45011cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): picked mon.noname-c con 0x7fa450132e10 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4500a1250 con 0x7fa450132150 2023-09-30T00:47:31.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4501313a0 con 0x7fa450098ca0 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa450120130 con 0x7fa450132e10 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa4556b2700 10 monclient(hunting): authenticate will time out at 1447.674234s 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 0x7fa45011c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 0x7fa45011c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39442/0 (socket says 172.21.15.6:39442) 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 -- 172.21.15.6:0/1965979357 learned_addr learned my addr 172.21.15.6:0/1965979357 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): get_auth_request con 0x7fa450132150 auth_method 0 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:31.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa450132e10 0x7fa45011cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): get_auth_request con 0x7fa450132e10 auth_method 0 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44effd700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450098ca0 0x7fa45012fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:31.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44effd700 10 monclient(hunting): get_auth_request con 0x7fa450098ca0 auth_method 0 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient(hunting): handle_auth_done global_id 4572 payload 824 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 -- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa450132e10 msgr2=0x7fa45011cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa450132e10 0x7fa45011cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 -- 172.21.15.6:0/1965979357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450098ca0 msgr2=0x7fa45012fe50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450098ca0 0x7fa45012fe50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:31.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 monclient: found mon.noname-a 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44e7fc700 1 -- 172.21.15.6:0/1965979357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa440009420 con 0x7fa450132150 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.376+0000 7fa44effd700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450098ca0 0x7fa45012fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.378+0000 7fa44e7fc700 1 -- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 msgr2=0x7fa45011c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.378+0000 7fa44e7fc700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 0x7fa45011c6f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa43800ea20 tx=0x7fa438009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43800eeb0 con 0x7fa450132150 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: dump: 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:31.393 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa43800ca60 con 0x7fa450132150 2023-09-30T00:47:31.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 5 monclient: authenticate success, global_id 4572 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43800ce20 con 0x7fa450132150 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 10 monclient: _renew_subs 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa45011fe20 con 0x7fa450132150 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa44d7fa700 4 set_mon_vals no callback set 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa446ffd700 10 monclient: dump: 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:31.395 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 10 monclient: _renew_subs 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.379+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa45012f280 con 0x7fa450132150 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa438013680 con 0x7fa450132150 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa4556b2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:31.396 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa4556b2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa4556b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa450130420 con 0x7fa450132150 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa446ffd700 1 --2- 172.21.15.6:0/1965979357 >> v2:172.21.15.22:6812/142603 conn(0x7fa4300357a0 0x7fa430037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa438047c10 con 0x7fa450132150 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa44effd700 1 --2- 172.21.15.6:0/1965979357 >> v2:172.21.15.22:6812/142603 conn(0x7fa4300357a0 0x7fa430037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.380+0000 7fa44effd700 10 monclient: get_auth_request con 0x7fa4300357a0 auth_method 0 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.383+0000 7fa44effd700 1 --2- 172.21.15.6:0/1965979357 >> v2:172.21.15.22:6812/142603 conn(0x7fa4300357a0 0x7fa430037c50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa4400030c0 tx=0x7fa440016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.385+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fa4380123d0 con 0x7fa450132150 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.385+0000 7fa446ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:31.397 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.385+0000 7fa446ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa4556b2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa4556b2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa4556b2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa450072b00 con 0x7fa450132150 2023-09-30T00:47:31.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa446ffd700 1 -- 172.21.15.6:0/1965979357 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fa438011350 con 0x7fa450132150 2023-09-30T00:47:31.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa446ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:31.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.630+0000 7fa446ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:31.632 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:31.632 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 >> v2:172.21.15.22:6812/142603 conn(0x7fa4300357a0 msgr2=0x7fa430037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 --2- 172.21.15.6:0/1965979357 >> v2:172.21.15.22:6812/142603 conn(0x7fa4300357a0 0x7fa430037c50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fa4400030c0 tx=0x7fa440016000 comp rx=0 tx=0).stop 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 10 monclient: shutdown 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 msgr2=0x7fa45011c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 0x7fa45011c6f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fa43800ea20 tx=0x7fa438009250 comp rx=0 tx=0).stop 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 shutdown_connections 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 --2- 172.21.15.6:0/1965979357 >> v2:172.21.15.22:6812/142603 conn(0x7fa4300357a0 0x7fa430037c50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa450132e10 0x7fa45011cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa450098ca0 0x7fa45012fe50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 --2- 172.21.15.6:0/1965979357 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa450132150 0x7fa45011c6f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 >> 172.21.15.6:0/1965979357 conn(0x7fa4500a54c0 msgr2=0x7fa4501309d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 shutdown_connections 2023-09-30T00:47:31.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:31.633+0000 7fa4556b2700 1 -- 172.21.15.6:0/1965979357 wait complete. 2023-09-30T00:47:34.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.786+0000 7f8dc9917700 10 monclient: get_monmap_and_config 2023-09-30T00:47:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.786+0000 7f8dc9917700 10 monclient: build_initial_monmap 2023-09-30T00:47:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.786+0000 7f8dc9917700 10 monclient: monmap: 2023-09-30T00:47:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:34.787 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:34.787887+0000 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:34.787887+0000 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:34.788 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 1 Processor -- start 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 1 -- start start 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient: init 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient: _add_conns ranks=[4,0,1,3,2] 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8dc4132150 0x7f8dc4132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient(hunting): picked mon.noname-c con 0x7f8dc4132150 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8dc4132e10 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8dc4098e70 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc4132520 con 0x7f8dc4132e10 2023-09-30T00:47:34.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8dc4096a80 con 0x7f8dc4098e70 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8dc40a0f80 con 0x7f8dc4132150 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.787+0000 7f8dc9917700 10 monclient(hunting): authenticate will time out at 1451.085936s 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc27fc700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8dc4132e10 0x7f8dc4098760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34990/0 (socket says 172.21.15.6:34990) 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc27fc700 1 -- 172.21.15.6:0/1713194188 learned_addr learned my addr 172.21.15.6:0/1713194188 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 1 --2- 172.21.15.6:0/1713194188 >> v2:172.21.15.22:3300/0 conn(0x7f8dc4132150 0x7f8dc4132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): get_auth_request con 0x7f8dc4132150 auth_method 0 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_done global_id 4573 payload 274 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 1 -- 172.21.15.6:0/1713194188 >> v1:172.21.15.22:6789/0 conn(0x7f8dc4098e70 legacy=0x7f8dc41394f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 1 -- 172.21.15.6:0/1713194188 >> v1:172.21.15.6:6789/0 conn(0x7f8dc4132e10 legacy=0x7f8dc4098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 1 monclient: found mon.noname-c 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:47:34.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.788+0000 7f8dc2ffd700 1 -- 172.21.15.6:0/1713194188 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dc4096d20 con 0x7f8dc4132150 2023-09-30T00:47:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.790+0000 7f8dc2ffd700 1 -- 172.21.15.6:0/1713194188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132150 msgr2=0x7f8dc4132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:34.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.790+0000 7f8dc2ffd700 1 --2- 172.21.15.6:0/1713194188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132150 0x7f8dc4132790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8db40098e0 tx=0x7f8db40091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4452b844146cc07 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 1 -- 172.21.15.6:0/1713194188 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db400ba30 con 0x7f8dc4132150 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: dump: 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:34.800 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 5 monclient: authenticate success, global_id 4573 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 1 -- 172.21.15.6:0/1713194188 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db400bb90 con 0x7f8dc4132150 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 1 -- 172.21.15.6:0/1713194188 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db4011420 con 0x7f8dc4132150 2023-09-30T00:47:34.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 4 set_mon_vals no callback set 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc1ffb700 10 monclient: dump: 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:34.802 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 10 monclient: shutdown 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 -- 172.21.15.6:0/1713194188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132150 msgr2=0x7f8dc4132790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1713194188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132150 0x7f8dc4132790 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8db40098e0 tx=0x7f8db40091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 -- 172.21.15.6:0/1713194188 shutdown_connections 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1713194188 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132150 0x7f8dc4132790 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 -- 172.21.15.6:0/1713194188 >> 172.21.15.6:0/1713194188 conn(0x7f8dc40a54c0 msgr2=0x7f8dc40a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 -- 172.21.15.6:0/1713194188 shutdown_connections 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.791+0000 7f8dc9917700 1 -- 172.21.15.6:0/1713194188 wait complete. 2023-09-30T00:47:34.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.792+0000 7f8dc9917700 10 monclient: build_initial_monmap 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.792+0000 7f8dc9917700 10 monclient: monmap: 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.792+0000 7f8dc9917700 1 Processor -- start 2023-09-30T00:47:34.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.792+0000 7f8dc9917700 1 -- start start 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.792+0000 7f8dc9917700 10 monclient: init 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8dc4098e70 0x7f8dc4133210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): picked mon.noname-c con 0x7f8dc4098e70 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 0x7f8dc4133750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): picked mon.noname-a con 0x7f8dc4132150 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132e10 0x7f8dc4133c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): picked mon.noname-b con 0x7f8dc4132e10 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4130060 con 0x7f8dc4132150 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc41365d0 con 0x7f8dc4132e10 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc4135400 con 0x7f8dc4098e70 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc9917700 10 monclient(hunting): authenticate will time out at 1451.091497s 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 0x7f8dc4133750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8dc4098e70 0x7f8dc4133210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 0x7f8dc4133750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:39452/0 (socket says 172.21.15.6:39452) 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8dc4098e70 0x7f8dc4133210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34056/0 (socket says 172.21.15.6:34056) 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 1 -- 172.21.15.6:0/1777952859 learned_addr learned my addr 172.21.15.6:0/1777952859 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): get_auth_request con 0x7f8dc4132150 auth_method 0 2023-09-30T00:47:34.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): get_auth_request con 0x7f8dc4098e70 auth_method 0 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132e10 0x7f8dc4133c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): get_auth_request con 0x7f8dc4132e10 auth_method 0 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc27fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.793+0000 7f8dc37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 10 monclient(hunting): handle_auth_done global_id 4575 payload 824 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 1 -- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8dc4098e70 msgr2=0x7f8dc4133210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8dc4098e70 0x7f8dc4133210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 1 -- 172.21.15.6:0/1777952859 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132e10 msgr2=0x7f8dc4133c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132e10 0x7f8dc4133c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:34.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 10 monclient: _finish_hunting 0 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 1 monclient: found mon.noname-a 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc27fc700 1 -- 172.21.15.6:0/1777952859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8db4009420 con 0x7f8dc4132150 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.794+0000 7f8dc37fe700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132e10 0x7f8dc4133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc27fc700 1 -- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 msgr2=0x7f8dc4133750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc27fc700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 0x7f8dc4133750 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8db800c5b0 tx=0x7f8db8007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8005d40 con 0x7f8dc4132150 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: dump: 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:34.809 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8db80070d0 con 0x7f8dc4132150 2023-09-30T00:47:34.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 5 monclient: authenticate success, global_id 4575 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 10 monclient: _renew_subs 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc41362f0 con 0x7f8dc4132150 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8db8007490 con 0x7f8dc4132150 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc17fa700 4 set_mon_vals no callback set 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8daaffd700 10 monclient: dump: 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.811 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 10 monclient: _renew_subs 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.796+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc4137510 con 0x7f8dc4132150 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8db801b3b0 con 0x7f8dc4132150 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8dc9917700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:34.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8dc9917700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8dc9917700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc40a4420 con 0x7f8dc4132150 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8daaffd700 1 --2- 172.21.15.6:0/1777952859 >> v2:172.21.15.22:6812/142603 conn(0x7f8db00357a0 0x7f8db0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8db8046da0 con 0x7f8dc4132150 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8dc2ffd700 1 --2- 172.21.15.6:0/1777952859 >> v2:172.21.15.22:6812/142603 conn(0x7f8db00357a0 0x7f8db0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.797+0000 7f8dc2ffd700 10 monclient: get_auth_request con 0x7f8db00357a0 auth_method 0 2023-09-30T00:47:34.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.800+0000 7f8dc2ffd700 1 --2- 172.21.15.6:0/1777952859 >> v2:172.21.15.22:6812/142603 conn(0x7f8db00357a0 0x7f8db0037c50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8db40030c0 tx=0x7f8db4016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:34.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.802+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8db801be60 con 0x7f8dc4132150 2023-09-30T00:47:34.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.802+0000 7f8daaffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:34.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:34.802+0000 7f8daaffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:35.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8dc9917700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:35.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8dc9917700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:35.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8dc9917700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:35.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8dc4072b00 con 0x7f8dc4132150 2023-09-30T00:47:35.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8daaffd700 1 -- 172.21.15.6:0/1777952859 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f8db8017020 con 0x7f8dc4132150 2023-09-30T00:47:35.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8daaffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:35.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.044+0000 7f8daaffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:35.046 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:35.046 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 >> v2:172.21.15.22:6812/142603 conn(0x7f8db00357a0 msgr2=0x7f8db0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1777952859 >> v2:172.21.15.22:6812/142603 conn(0x7f8db00357a0 0x7f8db0037c50 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8db40030c0 tx=0x7f8db4016000 comp rx=0 tx=0).stop 2023-09-30T00:47:35.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 10 monclient: shutdown 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 msgr2=0x7f8dc4133750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 0x7f8dc4133750 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8db800c5b0 tx=0x7f8db8007f90 comp rx=0 tx=0).stop 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 shutdown_connections 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1777952859 >> v2:172.21.15.22:6812/142603 conn(0x7f8db00357a0 0x7f8db0037c50 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8dc4132e10 0x7f8dc4133c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8dc4098e70 0x7f8dc4133210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 --2- 172.21.15.6:0/1777952859 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8dc4132150 0x7f8dc4133750 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.046+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 >> 172.21.15.6:0/1777952859 conn(0x7f8dc40a54c0 msgr2=0x7f8dc40974e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.047+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 shutdown_connections 2023-09-30T00:47:35.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:35.047+0000 7f8dc9917700 1 -- 172.21.15.6:0/1777952859 wait complete. 2023-09-30T00:47:38.061 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.199+0000 7f65241a7700 10 monclient: get_monmap_and_config 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.199+0000 7f65241a7700 10 monclient: build_initial_monmap 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.199+0000 7f65241a7700 10 monclient: monmap: 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:38.200566+0000 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:38.200566+0000 2023-09-30T00:47:38.200 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:38.201 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:38.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.199+0000 7f65241a7700 1 Processor -- start 2023-09-30T00:47:38.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 1 -- start start 2023-09-30T00:47:38.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient: init 2023-09-30T00:47:38.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient: _add_conns ranks=[2,1,0,3,4] 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 0x7f651c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient(hunting): picked mon.noname-b con 0x7f651c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f651c098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f651c098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c0a1000 con 0x7f651c098e90 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f651c12fc00 con 0x7f651c098a40 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c0a0a50 con 0x7f651c098670 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f65241a7700 10 monclient(hunting): authenticate will time out at 1454.498785s 2023-09-30T00:47:38.204 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6521f43700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 0x7f651c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6522744700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f651c098e90 0x7f651c12f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44320/0 (socket says 172.21.15.6:44320) 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6522744700 1 -- 172.21.15.6:0/2250676497 learned_addr learned my addr 172.21.15.6:0/2250676497 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6521f43700 10 monclient(hunting): get_auth_request con 0x7f651c098670 auth_method 0 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6521f43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6521f43700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.200+0000 7f6521f43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 10 monclient(hunting): handle_auth_done global_id 4577 payload 274 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 1 -- 172.21.15.6:0/2250676497 >> v1:172.21.15.22:6789/0 conn(0x7f651c098a40 legacy=0x7f651c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:38.205 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 1 -- 172.21.15.6:0/2250676497 >> v1:172.21.15.6:6789/0 conn(0x7f651c098e90 legacy=0x7f651c12f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:38.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 10 monclient: _finish_hunting 0 2023-09-30T00:47:38.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 1 monclient: found mon.noname-b 2023-09-30T00:47:38.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:38.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:38.206 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.201+0000 7f6521f43700 1 -- 172.21.15.6:0/2250676497 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f651c0a0730 con 0x7f651c098670 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6521f43700 1 -- 172.21.15.6:0/2250676497 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 msgr2=0x7f651c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6521f43700 1 --2- 172.21.15.6:0/2250676497 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 0x7f651c132d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6518009910 tx=0x7f65180091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b90e97b675dd0944 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 1 -- 172.21.15.6:0/2250676497 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f651800b9f0 con 0x7f651c098670 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: dump: 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:38.212 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: _finish_auth 0 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f65241a7700 5 monclient: authenticate success, global_id 4577 2023-09-30T00:47:38.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 1 -- 172.21.15.6:0/2250676497 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f651800bb50 con 0x7f651c098670 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f65241a7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f65241a7700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f65241a7700 4 set_mon_vals no callback set 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 1 -- 172.21.15.6:0/2250676497 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6518011420 con 0x7f651c098670 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.203+0000 7f6520f41700 10 monclient: dump: 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.214 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 10 monclient: shutdown 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 -- 172.21.15.6:0/2250676497 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 msgr2=0x7f651c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 --2- 172.21.15.6:0/2250676497 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 0x7f651c132d00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6518009910 tx=0x7f65180091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 -- 172.21.15.6:0/2250676497 shutdown_connections 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 --2- 172.21.15.6:0/2250676497 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098670 0x7f651c132d00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 -- 172.21.15.6:0/2250676497 >> 172.21.15.6:0/2250676497 conn(0x7f651c0a5480 msgr2=0x7f651c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 -- 172.21.15.6:0/2250676497 shutdown_connections 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.204+0000 7f65241a7700 1 -- 172.21.15.6:0/2250676497 wait complete. 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.205+0000 7f65241a7700 10 monclient: build_initial_monmap 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.205+0000 7f65241a7700 10 monclient: monmap: 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:38.216 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.205+0000 7f65241a7700 1 Processor -- start 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.205+0000 7f65241a7700 1 -- start start 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.205+0000 7f65241a7700 10 monclient: init 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f651c098670 0x7f651c141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): picked mon.noname-b con 0x7f651c098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 0x7f651c12c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): picked mon.noname-a con 0x7f651c098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098e90 0x7f651c12c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:38.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): picked mon.noname-c con 0x7f651c098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c142740 con 0x7f651c098a40 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f651c09e6e0 con 0x7f651c098670 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f651c12ed70 con 0x7f651c098e90 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f65241a7700 10 monclient(hunting): authenticate will time out at 1454.504592s 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 0x7f651c12c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 0x7f651c12c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42168/0 (socket says 172.21.15.6:42168) 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 1 -- 172.21.15.6:0/774844722 learned_addr learned my addr 172.21.15.6:0/774844722 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:38.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): get_auth_request con 0x7f651c098a40 auth_method 0 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f651c098670 0x7f651c141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098e90 0x7f651c12c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): get_auth_request con 0x7f651c098670 auth_method 0 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): get_auth_request con 0x7f651c098e90 auth_method 0 2023-09-30T00:47:38.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521742700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6521f43700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.206+0000 7f6522744700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 10 monclient(hunting): handle_auth_done global_id 4578 payload 824 2023-09-30T00:47:38.220 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 1 -- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098e90 msgr2=0x7f651c12c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098e90 0x7f651c12c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 1 -- 172.21.15.6:0/774844722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f651c098670 msgr2=0x7f651c141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f651c098670 0x7f651c141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 10 monclient: _finish_hunting 0 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 1 monclient: found mon.noname-a 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.207+0000 7f6521742700 1 -- 172.21.15.6:0/774844722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6518009420 con 0x7f651c098a40 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6521742700 1 -- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 msgr2=0x7f651c12c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6521742700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 0x7f651c12c200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c5d0 tx=0x7f650c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c0117a0 con 0x7f651c098a40 2023-09-30T00:47:38.221 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: dump: 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:38.222 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f650c011900 con 0x7f651c098a40 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 5 monclient: authenticate success, global_id 4578 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 10 monclient: _renew_subs 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f651c11c350 con 0x7f651c098a40 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f650c011cc0 con 0x7f651c098a40 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:38.223 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6511ffb700 10 monclient: dump: 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f6513fff700 4 set_mon_vals no callback set 2023-09-30T00:47:38.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 10 monclient: _renew_subs 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.209+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f651c141d80 con 0x7f651c098a40 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.210+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f650c012630 con 0x7f651c098a40 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.210+0000 7f65241a7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.210+0000 7f65241a7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.210+0000 7f65241a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.210+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f651c141fa0 con 0x7f651c098a40 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.211+0000 7f6511ffb700 1 --2- 172.21.15.6:0/774844722 >> v2:172.21.15.22:6812/142603 conn(0x7f65000357a0 0x7f6500037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.211+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f650c0475e0 con 0x7f651c098a40 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.211+0000 7f6521f43700 1 --2- 172.21.15.6:0/774844722 >> v2:172.21.15.22:6812/142603 conn(0x7f65000357a0 0x7f6500037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:38.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.211+0000 7f6521f43700 10 monclient: get_auth_request con 0x7f65000357a0 auth_method 0 2023-09-30T00:47:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.214+0000 7f6521f43700 1 --2- 172.21.15.6:0/774844722 >> v2:172.21.15.22:6812/142603 conn(0x7f65000357a0 0x7f6500037c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65180030c0 tx=0x7f6518016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.216+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f650c011e60 con 0x7f651c098a40 2023-09-30T00:47:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.216+0000 7f6511ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:38.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.216+0000 7f6511ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f65241a7700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f65241a7700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f65241a7700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f651c072b00 con 0x7f651c098a40 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f6511ffb700 1 -- 172.21.15.6:0/774844722 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f650c0102e0 con 0x7f651c098a40 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f6511ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:38.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.461+0000 7f6511ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:38.463 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:38.463 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:38.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 >> v2:172.21.15.22:6812/142603 conn(0x7f65000357a0 msgr2=0x7f6500037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:38.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 --2- 172.21.15.6:0/774844722 >> v2:172.21.15.22:6812/142603 conn(0x7f65000357a0 0x7f6500037c50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f65180030c0 tx=0x7f6518016000 comp rx=0 tx=0).stop 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 10 monclient: shutdown 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 msgr2=0x7f651c12c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 0x7f651c12c200 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f650c00c5d0 tx=0x7f650c007f60 comp rx=0 tx=0).stop 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 shutdown_connections 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 --2- 172.21.15.6:0/774844722 >> v2:172.21.15.22:6812/142603 conn(0x7f65000357a0 0x7f6500037c50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f651c098e90 0x7f651c12c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f651c098670 0x7f651c141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 --2- 172.21.15.6:0/774844722 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f651c098a40 0x7f651c12c200 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 >> 172.21.15.6:0/774844722 conn(0x7f651c0a5480 msgr2=0x7f651c12e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.463+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 shutdown_connections 2023-09-30T00:47:38.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:38.464+0000 7f65241a7700 1 -- 172.21.15.6:0/774844722 wait complete. 2023-09-30T00:47:41.478 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.616+0000 7f3e2a582700 10 monclient: get_monmap_and_config 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.616+0000 7f3e2a582700 10 monclient: build_initial_monmap 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.616+0000 7f3e2a582700 10 monclient: monmap: 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:41.617498+0000 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:41.617498+0000 2023-09-30T00:47:41.617 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:41.618 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:41.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 1 Processor -- start 2023-09-30T00:47:41.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 1 -- start start 2023-09-30T00:47:41.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient: init 2023-09-30T00:47:41.620 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3e24098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3e24132d00 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f3e24138d40 0x7f3e2413b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient(hunting): picked mon.noname-c con 0x7f3e24138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240a1000 con 0x7f3e24098670 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e240a0730 con 0x7f3e24132d00 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e24099150 con 0x7f3e24138d40 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e2a582700 10 monclient(hunting): authenticate will time out at 1457.915690s 2023-09-30T00:47:41.621 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e23fff700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f3e24098670 0x7f3e24098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44334/0 (socket says 172.21.15.6:44334) 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e23fff700 1 -- 172.21.15.6:0/901913430 learned_addr learned my addr 172.21.15.6:0/901913430 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.617+0000 7f3e28b1f700 1 --2- 172.21.15.6:0/901913430 >> v2:172.21.15.22:3300/0 conn(0x7f3e24138d40 0x7f3e2413b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): get_auth_request con 0x7f3e24138d40 auth_method 0 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_done global_id 4582 payload 274 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 1 -- 172.21.15.6:0/901913430 >> v1:172.21.15.22:6789/0 conn(0x7f3e24132d00 legacy=0x7f3e24137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:41.622 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 1 -- 172.21.15.6:0/901913430 >> v1:172.21.15.6:6789/0 conn(0x7f3e24098670 legacy=0x7f3e24098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient: _finish_hunting 0 2023-09-30T00:47:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 1 monclient: found mon.noname-c 2023-09-30T00:47:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:47:41.623 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.618+0000 7f3e28b1f700 1 -- 172.21.15.6:0/901913430 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e24068df0 con 0x7f3e24138d40 2023-09-30T00:47:41.628 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e28b1f700 1 -- 172.21.15.6:0/901913430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24138d40 msgr2=0x7f3e2413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e28b1f700 1 --2- 172.21.15.6:0/901913430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24138d40 0x7f3e2413b120 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009e60 tx=0x7f3e1c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74d74d4cf2cc7f5a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 1 -- 172.21.15.6:0/901913430 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c00e070 con 0x7f3e24138d40 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: dump: 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:41.629 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 1 -- 172.21.15.6:0/901913430 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c012430 con 0x7f3e24138d40 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e2a582700 5 monclient: authenticate success, global_id 4582 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e2a582700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e2a582700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:41.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 1 -- 172.21.15.6:0/901913430 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0127d0 con 0x7f3e24138d40 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e2a582700 4 set_mon_vals no callback set 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.620+0000 7f3e22ffd700 10 monclient: dump: 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:41.631 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 10 monclient: shutdown 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 -- 172.21.15.6:0/901913430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24138d40 msgr2=0x7f3e2413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 --2- 172.21.15.6:0/901913430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24138d40 0x7f3e2413b120 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3e1c009e60 tx=0x7f3e1c009780 comp rx=0 tx=0).stop 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 -- 172.21.15.6:0/901913430 shutdown_connections 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 --2- 172.21.15.6:0/901913430 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24138d40 0x7f3e2413b120 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 -- 172.21.15.6:0/901913430 >> 172.21.15.6:0/901913430 conn(0x7f3e240a5480 msgr2=0x7f3e240a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 -- 172.21.15.6:0/901913430 shutdown_connections 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 1 -- 172.21.15.6:0/901913430 wait complete. 2023-09-30T00:47:41.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 10 monclient: build_initial_monmap 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.621+0000 7f3e2a582700 10 monclient: monmap: 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.622+0000 7f3e2a582700 1 Processor -- start 2023-09-30T00:47:41.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.622+0000 7f3e2a582700 1 -- start start 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.622+0000 7f3e2a582700 10 monclient: init 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.622+0000 7f3e2a582700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.622+0000 7f3e2a582700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 0x7f3e241415c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): picked mon.noname-a con 0x7f3e24098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24132d00 0x7f3e24138350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): picked mon.noname-b con 0x7f3e24132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 0x7f3e2413f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): picked mon.noname-c con 0x7f3e24138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2412f420 con 0x7f3e24098670 2023-09-30T00:47:41.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e24130320 con 0x7f3e24132d00 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e2412e0e0 con 0x7f3e24138d40 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e2a582700 10 monclient(hunting): authenticate will time out at 1457.921073s 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 0x7f3e241415c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 0x7f3e2413f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 0x7f3e241415c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42170/0 (socket says 172.21.15.6:42170) 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): get_auth_request con 0x7f3e24138d40 auth_method 0 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 1 -- 172.21.15.6:0/1003672780 learned_addr learned my addr 172.21.15.6:0/1003672780 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:41.635 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): get_auth_request con 0x7f3e24098670 auth_method 0 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:41.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e237fe700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24132d00 0x7f3e24138350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e237fe700 10 monclient(hunting): get_auth_request con 0x7f3e24132d00 auth_method 0 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e237fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e237fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e237fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient(hunting): handle_auth_done global_id 4583 payload 824 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 -- 172.21.15.6:0/1003672780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24132d00 msgr2=0x7f3e24138350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24132d00 0x7f3e24138350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 -- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 msgr2=0x7f3e241415c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 0x7f3e241415c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient: _finish_hunting 0 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 monclient: found mon.noname-c 2023-09-30T00:47:41.637 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e28b1f700 1 -- 172.21.15.6:0/1003672780 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e1c009a20 con 0x7f3e24138d40 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.623+0000 7f3e23fff700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 0x7f3e241415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.625+0000 7f3e28b1f700 1 -- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 msgr2=0x7f3e2413f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.625+0000 7f3e28b1f700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 0x7f3e2413f4c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e2409fe60 tx=0x7f3e1c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c026430 con 0x7f3e24138d40 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: dump: 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:41.638 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: _finish_auth 0 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 5 monclient: authenticate success, global_id 4583 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e1c021070 con 0x7f3e24138d40 2023-09-30T00:47:41.639 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 10 monclient: _renew_subs 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e1c0267e0 con 0x7f3e24138d40 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e24133550 con 0x7f3e24138d40 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e227fc700 4 set_mon_vals no callback set 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 10 monclient: dump: 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:41.640 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 10 monclient: _renew_subs 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e2412e760 con 0x7f3e24138d40 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.626+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3e1c023030 con 0x7f3e24138d40 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e2a582700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e2a582700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:41.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e2a582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:41.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e24072b00 con 0x7f3e24138d40 2023-09-30T00:47:41.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e0bfff700 1 --2- 172.21.15.6:0/1003672780 >> v2:172.21.15.22:6812/142603 conn(0x7f3e100357a0 0x7f3e10037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:41.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3e1c049c10 con 0x7f3e24138d40 2023-09-30T00:47:41.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e23fff700 1 --2- 172.21.15.6:0/1003672780 >> v2:172.21.15.22:6812/142603 conn(0x7f3e100357a0 0x7f3e10037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:41.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.627+0000 7f3e23fff700 10 monclient: get_auth_request con 0x7f3e100357a0 auth_method 0 2023-09-30T00:47:41.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.630+0000 7f3e23fff700 1 --2- 172.21.15.6:0/1003672780 >> v2:172.21.15.22:6812/142603 conn(0x7f3e100357a0 0x7f3e10037c50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3e140052e0 tx=0x7f3e14004a80 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:41.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.632+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f3e1c01d420 con 0x7f3e24138d40 2023-09-30T00:47:41.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.632+0000 7f3e0bfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:41.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.632+0000 7f3e0bfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e2a582700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e2a582700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e2a582700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:41.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3e24072b00 con 0x7f3e24138d40 2023-09-30T00:47:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e0bfff700 1 -- 172.21.15.6:0/1003672780 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f3e1c0253d0 con 0x7f3e24138d40 2023-09-30T00:47:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e0bfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:41.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.882+0000 7f3e0bfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:41.884 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:41.884 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:41.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 >> v2:172.21.15.22:6812/142603 conn(0x7f3e100357a0 msgr2=0x7f3e10037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 --2- 172.21.15.6:0/1003672780 >> v2:172.21.15.22:6812/142603 conn(0x7f3e100357a0 0x7f3e10037c50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3e140052e0 tx=0x7f3e14004a80 comp rx=0 tx=0).stop 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 10 monclient: shutdown 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 msgr2=0x7f3e2413f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 0x7f3e2413f4c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3e2409fe60 tx=0x7f3e1c018000 comp rx=0 tx=0).stop 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 shutdown_connections 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 --2- 172.21.15.6:0/1003672780 >> v2:172.21.15.22:6812/142603 conn(0x7f3e100357a0 0x7f3e10037c50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3e24138d40 0x7f3e2413f4c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3e24098670 0x7f3e241415c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 --2- 172.21.15.6:0/1003672780 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3e24132d00 0x7f3e24138350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:41.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 >> 172.21.15.6:0/1003672780 conn(0x7f3e240a5480 msgr2=0x7f3e240a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:41.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 shutdown_connections 2023-09-30T00:47:41.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:41.885+0000 7f3e2a582700 1 -- 172.21.15.6:0/1003672780 wait complete. 2023-09-30T00:47:44.899 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:45.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.035+0000 7f7381c6e700 10 monclient: get_monmap_and_config 2023-09-30T00:47:45.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.035+0000 7f7381c6e700 10 monclient: build_initial_monmap 2023-09-30T00:47:45.036 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.035+0000 7f7381c6e700 10 monclient: monmap: 2023-09-30T00:47:45.036 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:45.037117+0000 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:45.037117+0000 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:45.037 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:45.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.036+0000 7f7381c6e700 1 Processor -- start 2023-09-30T00:47:45.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.036+0000 7f7381c6e700 1 -- start start 2023-09-30T00:47:45.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.036+0000 7f7381c6e700 10 monclient: init 2023-09-30T00:47:45.041 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient: _add_conns ranks=[2,0,4,1,3] 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): picked mon.noname-b con 0x7f737c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f737c098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f737c098e90 0x7f737c12f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): picked mon.noname-c con 0x7f737c098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f737c0a1000 con 0x7f737c098a40 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f737c0a0a50 con 0x7f737c098e90 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c0a0730 con 0x7f737c098670 2023-09-30T00:47:45.042 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f7381c6e700 10 monclient(hunting): authenticate will time out at 1461.335338s 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737affd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f737c098a40 0x7f737c12ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:44348/0 (socket says 172.21.15.6:44348) 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737affd700 1 -- 172.21.15.6:0/1030016902 learned_addr learned my addr 172.21.15.6:0/1030016902 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 1 --2- 172.21.15.6:0/1030016902 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): get_auth_request con 0x7f737c098670 auth_method 0 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:45.043 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737bfff700 1 --2- 172.21.15.6:0/1030016902 >> v2:172.21.15.22:3300/0 conn(0x7f737c098e90 0x7f737c12f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737bfff700 10 monclient(hunting): get_auth_request con 0x7f737c098e90 auth_method 0 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient(hunting): handle_auth_done global_id 4586 payload 274 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 1 -- 172.21.15.6:0/1030016902 >> v2:172.21.15.22:3300/0 conn(0x7f737c098e90 msgr2=0x7f737c12f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 1 --2- 172.21.15.6:0/1030016902 >> v2:172.21.15.22:3300/0 conn(0x7f737c098e90 0x7f737c12f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 1 -- 172.21.15.6:0/1030016902 >> v1:172.21.15.6:6789/0 conn(0x7f737c098a40 legacy=0x7f737c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 1 monclient: found mon.noname-b 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:45.044 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737b7fe700 1 -- 172.21.15.6:0/1030016902 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f737c068df0 con 0x7f737c098670 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.037+0000 7f737bfff700 1 --2- 172.21.15.6:0/1030016902 >> v2:172.21.15.22:3300/0 conn(0x7f737c098e90 0x7f737c12f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737b7fe700 1 -- 172.21.15.6:0/1030016902 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 msgr2=0x7f737c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737b7fe700 1 --2- 172.21.15.6:0/1030016902 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c132d00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f736c0098e0 tx=0x7f736c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=596f07adab07a4a2 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 1 -- 172.21.15.6:0/1030016902 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c009e80 con 0x7f737c098670 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: dump: 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.045 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 1 -- 172.21.15.6:0/1030016902 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f736c00ba30 con 0x7f737c098670 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 5 monclient: authenticate success, global_id 4586 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:45.046 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 1 -- 172.21.15.6:0/1030016902 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f736c00bdb0 con 0x7f737c098670 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 4 set_mon_vals no callback set 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f737a7fc700 10 monclient: dump: 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:45.047 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 10 monclient: shutdown 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 1 -- 172.21.15.6:0/1030016902 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 msgr2=0x7f737c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.040+0000 7f7381c6e700 1 --2- 172.21.15.6:0/1030016902 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c132d00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f736c0098e0 tx=0x7f736c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 1 -- 172.21.15.6:0/1030016902 shutdown_connections 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 1 --2- 172.21.15.6:0/1030016902 >> v2:172.21.15.22:3300/0 conn(0x7f737c098e90 0x7f737c12f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 1 --2- 172.21.15.6:0/1030016902 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c132d00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 1 -- 172.21.15.6:0/1030016902 >> 172.21.15.6:0/1030016902 conn(0x7f737c0a5480 msgr2=0x7f737c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:45.048 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 1 -- 172.21.15.6:0/1030016902 shutdown_connections 2023-09-30T00:47:45.049 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 1 -- 172.21.15.6:0/1030016902 wait complete. 2023-09-30T00:47:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 10 monclient: build_initial_monmap 2023-09-30T00:47:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.041+0000 7f7381c6e700 10 monclient: monmap: 2023-09-30T00:47:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:45.053 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 Processor -- start 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 -- start start 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient: init 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:45.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c09ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): picked mon.noname-c con 0x7f737c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 0x7f737c12e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): picked mon.noname-a con 0x7f737c098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f737c098e90 0x7f737c12ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): picked mon.noname-b con 0x7f737c098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c09e050 con 0x7f737c098a40 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f737c09dd70 con 0x7f737c098e90 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f737c09e330 con 0x7f737c098670 2023-09-30T00:47:45.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f7381c6e700 10 monclient(hunting): authenticate will time out at 1461.340810s 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c09ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c09ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:45152/0 (socket says 172.21.15.6:45152) 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 0x7f737c12e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 1 -- 172.21.15.6:0/3319981260 learned_addr learned my addr 172.21.15.6:0/3319981260 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 10 monclient(hunting): get_auth_request con 0x7f737c098670 auth_method 0 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737affd700 10 monclient(hunting): get_auth_request con 0x7f737c098a40 auth_method 0 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:45.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.042+0000 7f737affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f737c098e90 0x7f737c12ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): get_auth_request con 0x7f737c098e90 auth_method 0 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:45.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 10 monclient(hunting): handle_auth_done global_id 4584 payload 824 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 1 -- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 msgr2=0x7f737c09ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c09ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 1 -- 172.21.15.6:0/3319981260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f737c098e90 msgr2=0x7f737c12ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f737c098e90 0x7f737c12ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 1 monclient: found mon.noname-a 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:45.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737affd700 1 -- 172.21.15.6:0/3319981260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f736c009420 con 0x7f737c098a40 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.043+0000 7f737b7fe700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c09ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f737affd700 1 -- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 msgr2=0x7f737c12e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f737affd700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 0x7f737c12e6c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f737000c530 tx=0x7f7370007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370005d40 con 0x7f737c098a40 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: dump: 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.059 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73700070d0 con 0x7f737c098a40 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 5 monclient: authenticate success, global_id 4584 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:45.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7370007490 con 0x7f737c098a40 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 10 monclient: _renew_subs 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f737c12f900 con 0x7f737c098a40 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7379ffb700 4 set_mon_vals no callback set 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f73637fe700 10 monclient: dump: 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:45.061 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 10 monclient: _renew_subs 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.045+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f737c09f2c0 con 0x7f737c098a40 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f737001b3b0 con 0x7f737c098a40 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f7381c6e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f7381c6e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f7381c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:45.062 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f737c09ce80 con 0x7f737c098a40 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f73637fe700 1 --2- 172.21.15.6:0/3319981260 >> v2:172.21.15.22:6812/142603 conn(0x7f73680357a0 0x7f7368037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.046+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f7370046d40 con 0x7f737c098a40 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.047+0000 7f737b7fe700 1 --2- 172.21.15.6:0/3319981260 >> v2:172.21.15.22:6812/142603 conn(0x7f73680357a0 0x7f7368037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.047+0000 7f737b7fe700 10 monclient: get_auth_request con 0x7f73680357a0 auth_method 0 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.049+0000 7f737b7fe700 1 --2- 172.21.15.6:0/3319981260 >> v2:172.21.15.22:6812/142603 conn(0x7f73680357a0 0x7f7368037c50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f736c0030c0 tx=0x7f736c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.051+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f737001be60 con 0x7f737c098a40 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.051+0000 7f73637fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:45.063 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.051+0000 7f73637fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f7381c6e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f7381c6e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f7381c6e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:45.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f737c072b00 con 0x7f737c098a40 2023-09-30T00:47:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f73637fe700 1 -- 172.21.15.6:0/3319981260 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f7370017070 con 0x7f737c098a40 2023-09-30T00:47:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f73637fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:45.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.297+0000 7f73637fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:45.299 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:45.299 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:45.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 >> v2:172.21.15.22:6812/142603 conn(0x7f73680357a0 msgr2=0x7f7368037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 1 --2- 172.21.15.6:0/3319981260 >> v2:172.21.15.22:6812/142603 conn(0x7f73680357a0 0x7f7368037c50 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f736c0030c0 tx=0x7f736c016000 comp rx=0 tx=0).stop 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 10 monclient: shutdown 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 msgr2=0x7f737c12e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 0x7f737c12e6c0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f737000c530 tx=0x7f7370007f90 comp rx=0 tx=0).stop 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.299+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 shutdown_connections 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 --2- 172.21.15.6:0/3319981260 >> v2:172.21.15.22:6812/142603 conn(0x7f73680357a0 0x7f7368037c50 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f737c098e90 0x7f737c12ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f737c098670 0x7f737c09ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 --2- 172.21.15.6:0/3319981260 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f737c098a40 0x7f737c12e6c0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 >> 172.21.15.6:0/3319981260 conn(0x7f737c0a5480 msgr2=0x7f737c133b30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:45.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 shutdown_connections 2023-09-30T00:47:45.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:45.300+0000 7f7381c6e700 1 -- 172.21.15.6:0/3319981260 wait complete. 2023-09-30T00:47:48.314 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:48.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.454+0000 7f98d2b51700 10 monclient: get_monmap_and_config 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.454+0000 7f98d2b51700 10 monclient: build_initial_monmap 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.454+0000 7f98d2b51700 10 monclient: monmap: 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:48.454284+0000 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:48.454284+0000 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:48.454 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:48.455 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:48.455 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:48.455 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:48.455 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:48.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.454+0000 7f98d2b51700 1 Processor -- start 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.454+0000 7f98d2b51700 1 -- start start 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.454+0000 7f98d2b51700 10 monclient: init 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient: _add_conns ranks=[2,1,4,0,3] 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 0x7f98cc132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): picked mon.noname-b con 0x7f98cc098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f98cc098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f98cc098e90 0x7f98cc12f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): picked mon.noname-c con 0x7f98cc098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0a1000 con 0x7f98cc098a40 2023-09-30T00:47:48.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0a0a50 con 0x7f98cc098e90 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0a0730 con 0x7f98cc098670 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d2b51700 10 monclient(hunting): authenticate will time out at 1464.752477s 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 0x7f98cc132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 0x7f98cc132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41292/0 (socket says 172.21.15.6:41292) 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 1 -- 172.21.15.6:0/3006791921 learned_addr learned my addr 172.21.15.6:0/3006791921 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): get_auth_request con 0x7f98cc098670 auth_method 0 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:48.460 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d08ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 1 --2- 172.21.15.6:0/3006791921 >> v2:172.21.15.22:3300/0 conn(0x7f98cc098e90 0x7f98cc12f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): get_auth_request con 0x7f98cc098e90 auth_method 0 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.455+0000 7f98d10ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 10 monclient(hunting): handle_auth_done global_id 4592 payload 274 2023-09-30T00:47:48.461 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 1 -- 172.21.15.6:0/3006791921 >> v2:172.21.15.22:3300/0 conn(0x7f98cc098e90 msgr2=0x7f98cc12f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 1 --2- 172.21.15.6:0/3006791921 >> v2:172.21.15.22:3300/0 conn(0x7f98cc098e90 0x7f98cc12f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 1 -- 172.21.15.6:0/3006791921 >> v1:172.21.15.22:6789/0 conn(0x7f98cc098a40 legacy=0x7f98cc12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 10 monclient: _finish_hunting 0 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 1 monclient: found mon.noname-b 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.456+0000 7f98d08ed700 1 -- 172.21.15.6:0/3006791921 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc068df0 con 0x7f98cc098670 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98d08ed700 1 -- 172.21.15.6:0/3006791921 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 msgr2=0x7f98cc132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98d08ed700 1 --2- 172.21.15.6:0/3006791921 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 0x7f98cc132d00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0099b0 tx=0x7f98bc009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb2ace00a1cd2e88 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 1 -- 172.21.15.6:0/3006791921 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc009ea0 con 0x7f98cc098670 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:48.462 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: dump: 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:48.463 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: _finish_auth 0 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 1 -- 172.21.15.6:0/3006791921 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98bc00ba30 con 0x7f98cc098670 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98d2b51700 5 monclient: authenticate success, global_id 4592 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98d2b51700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98d2b51700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 1 -- 172.21.15.6:0/3006791921 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98bc00bdf0 con 0x7f98cc098670 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98d2b51700 4 set_mon_vals no callback set 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:48.464 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.458+0000 7f98cb7fe700 10 monclient: dump: 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 10 monclient: shutdown 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:48.465 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 -- 172.21.15.6:0/3006791921 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 msgr2=0x7f98cc132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 --2- 172.21.15.6:0/3006791921 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 0x7f98cc132d00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f98bc0099b0 tx=0x7f98bc009250 comp rx=0 tx=0).stop 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 -- 172.21.15.6:0/3006791921 shutdown_connections 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 --2- 172.21.15.6:0/3006791921 >> v2:172.21.15.22:3300/0 conn(0x7f98cc098e90 0x7f98cc12f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 --2- 172.21.15.6:0/3006791921 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098670 0x7f98cc132d00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 -- 172.21.15.6:0/3006791921 >> 172.21.15.6:0/3006791921 conn(0x7f98cc0a5480 msgr2=0x7f98cc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 -- 172.21.15.6:0/3006791921 shutdown_connections 2023-09-30T00:47:48.466 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 1 -- 172.21.15.6:0/3006791921 wait complete. 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 10 monclient: build_initial_monmap 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.459+0000 7f98d2b51700 10 monclient: monmap: 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:48.471 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.460+0000 7f98d2b51700 1 Processor -- start 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.460+0000 7f98d2b51700 1 -- start start 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.460+0000 7f98d2b51700 10 monclient: init 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 0x7f98cc09ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): picked mon.noname-a con 0x7f98cc098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:48.472 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f98cc098a40 0x7f98cc12e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): picked mon.noname-b con 0x7f98cc098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 0x7f98cc12e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): picked mon.noname-c con 0x7f98cc098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc12fc80 con 0x7f98cc098670 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc09e2e0 con 0x7f98cc098a40 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc09e000 con 0x7f98cc098e90 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d2b51700 10 monclient(hunting): authenticate will time out at 1464.758402s 2023-09-30T00:47:48.473 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 0x7f98cc09ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 0x7f98cc09ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46882/0 (socket says 172.21.15.6:46882) 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 1 -- 172.21.15.6:0/4142430150 learned_addr learned my addr 172.21.15.6:0/4142430150 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 0x7f98cc12e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): get_auth_request con 0x7f98cc098670 auth_method 0 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): get_auth_request con 0x7f98cc098e90 auth_method 0 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:48.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98cbfff700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f98cc098a40 0x7f98cc12e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98cbfff700 10 monclient(hunting): get_auth_request con 0x7f98cc098a40 auth_method 0 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98cbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98cbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98cbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:48.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient(hunting): handle_auth_done global_id 4595 payload 824 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 -- 172.21.15.6:0/4142430150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f98cc098a40 msgr2=0x7f98cc12e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f98cc098a40 0x7f98cc12e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 -- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 msgr2=0x7f98cc09ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 0x7f98cc09ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient: _finish_hunting 0 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 monclient: found mon.noname-c 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d10ee700 1 -- 172.21.15.6:0/4142430150 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98bc0094f0 con 0x7f98cc098e90 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98cbfff700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f98cc098a40 0x7f98cc12e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.461+0000 7f98d08ed700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 0x7f98cc09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:48.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d10ee700 1 -- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 msgr2=0x7f98cc12e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d10ee700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 0x7f98cc12e660 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009e20 tx=0x7f98c400b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c400cd60 con 0x7f98cc098e90 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: dump: 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:48.477 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 5 monclient: authenticate success, global_id 4595 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c400f040 con 0x7f98cc098e90 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f98c400cee0 con 0x7f98cc098e90 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:48.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98caffd700 4 set_mon_vals no callback set 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98c8ff9700 10 monclient: dump: 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:48.479 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 10 monclient: _renew_subs 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc09e5c0 con 0x7f98cc098e90 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 10 monclient: _renew_subs 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.464+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc09f4c0 con 0x7f98cc098e90 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98d2b51700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98d2b51700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98d2b51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:48.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc072b00 con 0x7f98cc098e90 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f98c40148e0 con 0x7f98cc098e90 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98c8ff9700 1 --2- 172.21.15.6:0/4142430150 >> v2:172.21.15.22:6812/142603 conn(0x7f98ac0357d0 0x7f98ac037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.465+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f98c4048560 con 0x7f98cc098e90 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.466+0000 7f98d08ed700 1 --2- 172.21.15.6:0/4142430150 >> v2:172.21.15.22:6812/142603 conn(0x7f98ac0357d0 0x7f98ac037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.466+0000 7f98d08ed700 10 monclient: get_auth_request con 0x7f98ac0357d0 auth_method 0 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.468+0000 7f98d08ed700 1 --2- 172.21.15.6:0/4142430150 >> v2:172.21.15.22:6812/142603 conn(0x7f98ac0357d0 0x7f98ac037c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009470 tx=0x7f98bc016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.470+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f98c4012b70 con 0x7f98cc098e90 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.470+0000 7f98c8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:48.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.470+0000 7f98c8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:48.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.715+0000 7f98d2b51700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:48.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.715+0000 7f98d2b51700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:48.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.715+0000 7f98d2b51700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:48.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.715+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f98cc063920 con 0x7f98cc098e90 2023-09-30T00:47:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.715+0000 7f98c8ff9700 1 -- 172.21.15.6:0/4142430150 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f98c4018020 con 0x7f98cc098e90 2023-09-30T00:47:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.716+0000 7f98c8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:48.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.716+0000 7f98c8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:48.716 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:48.716 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 >> v2:172.21.15.22:6812/142603 conn(0x7f98ac0357d0 msgr2=0x7f98ac037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 --2- 172.21.15.6:0/4142430150 >> v2:172.21.15.22:6812/142603 conn(0x7f98ac0357d0 0x7f98ac037c80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f98bc009470 tx=0x7f98bc016000 comp rx=0 tx=0).stop 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 10 monclient: shutdown 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 msgr2=0x7f98cc12e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 0x7f98cc12e660 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f98c4009e20 tx=0x7f98c400b040 comp rx=0 tx=0).stop 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 shutdown_connections 2023-09-30T00:47:48.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 --2- 172.21.15.6:0/4142430150 >> v2:172.21.15.22:6812/142603 conn(0x7f98ac0357d0 0x7f98ac037c80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f98cc098e90 0x7f98cc12e660 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f98cc098670 0x7f98cc09ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 --2- 172.21.15.6:0/4142430150 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f98cc098a40 0x7f98cc12e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 >> 172.21.15.6:0/4142430150 conn(0x7f98cc0a5480 msgr2=0x7f98cc133ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 shutdown_connections 2023-09-30T00:47:48.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:48.718+0000 7f98d2b51700 1 -- 172.21.15.6:0/4142430150 wait complete. 2023-09-30T00:47:51.731 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.872+0000 7f1445753700 10 monclient: get_monmap_and_config 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.872+0000 7f1445753700 10 monclient: build_initial_monmap 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.872+0000 7f1445753700 10 monclient: monmap: 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:51.872586+0000 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:51.872586+0000 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:51.872 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:51.873 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:51.873 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:51.879 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.872+0000 7f1445753700 1 Processor -- start 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 -- start start 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient: init 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient: _add_conns ranks=[2,3,4,0,1] 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f1440132770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): picked mon.noname-b con 0x7f14401323a0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 0x7f14400989c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): picked mon.noname-a con 0x7f1440098570 addr v2:172.21.15.6:3300/0 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1440068df0 0x7f1440069250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): picked mon.noname-c con 0x7f1440068df0 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.880 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1440096a80 con 0x7f1440098570 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1440096d60 con 0x7f1440068df0 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14400a1c30 con 0x7f14401323a0 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f1445753700 10 monclient(hunting): authenticate will time out at 1468.170634s 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 0x7f14400989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f1440132770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 0x7f14400989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46906/0 (socket says 172.21.15.6:46906) 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 1 -- 172.21.15.6:0/3518915252 learned_addr learned my addr 172.21.15.6:0/3518915252 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f1440132770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41316/0 (socket says 172.21.15.6:41316) 2023-09-30T00:47:51.881 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): get_auth_request con 0x7f1440098570 auth_method 0 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): get_auth_request con 0x7f14401323a0 auth_method 0 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143f7fe700 1 --2- 172.21.15.6:0/3518915252 >> v2:172.21.15.22:3300/0 conn(0x7f1440068df0 0x7f1440069250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.882 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143f7fe700 10 monclient(hunting): get_auth_request con 0x7f1440068df0 auth_method 0 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.873+0000 7f143f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 10 monclient(hunting): handle_auth_done global_id 4598 payload 274 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 1 -- 172.21.15.6:0/3518915252 >> v2:172.21.15.22:3300/0 conn(0x7f1440068df0 msgr2=0x7f1440069250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 1 --2- 172.21.15.6:0/3518915252 >> v2:172.21.15.22:3300/0 conn(0x7f1440068df0 0x7f1440069250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.883 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 1 -- 172.21.15.6:0/3518915252 >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 msgr2=0x7f14400989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 1 --2- 172.21.15.6:0/3518915252 >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 0x7f14400989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 1 monclient: found mon.noname-b 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143effd700 1 -- 172.21.15.6:0/3518915252 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14400a0a20 con 0x7f14401323a0 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.874+0000 7f143e7fc700 1 --2- 172.21.15.6:0/3518915252 >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 0x7f14400989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143effd700 1 -- 172.21.15.6:0/3518915252 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 msgr2=0x7f1440132770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143effd700 1 --2- 172.21.15.6:0/3518915252 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f1440132770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14400a09d0 tx=0x7f14300091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad8cd8013d7478f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 1 -- 172.21.15.6:0/3518915252 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430011840 con 0x7f14401323a0 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.884 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: dump: 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:51.885 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f1445753700 5 monclient: authenticate success, global_id 4598 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f1445753700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 1 -- 172.21.15.6:0/3518915252 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14300119c0 con 0x7f14401323a0 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f1445753700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f1445753700 4 set_mon_vals no callback set 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 1 -- 172.21.15.6:0/3518915252 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1430011d80 con 0x7f14401323a0 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.876+0000 7f143dffb700 10 monclient: dump: 2023-09-30T00:47:51.886 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 10 monclient: shutdown 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:51.887 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 -- 172.21.15.6:0/3518915252 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 msgr2=0x7f1440132770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 --2- 172.21.15.6:0/3518915252 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f1440132770 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f14400a09d0 tx=0x7f14300091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 -- 172.21.15.6:0/3518915252 shutdown_connections 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 --2- 172.21.15.6:0/3518915252 >> v2:172.21.15.22:3300/0 conn(0x7f1440068df0 0x7f1440069250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 --2- 172.21.15.6:0/3518915252 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f1440132770 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 --2- 172.21.15.6:0/3518915252 >> v2:172.21.15.6:3300/0 conn(0x7f1440098570 0x7f14400989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 -- 172.21.15.6:0/3518915252 >> 172.21.15.6:0/3518915252 conn(0x7f14400a5480 msgr2=0x7f14400a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 -- 172.21.15.6:0/3518915252 shutdown_connections 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 1 -- 172.21.15.6:0/3518915252 wait complete. 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 10 monclient: build_initial_monmap 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.877+0000 7f1445753700 10 monclient: monmap: 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:51.888 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 Processor -- start 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 -- start start 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient: init 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:51.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1440068df0 0x7f1440141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): picked mon.noname-b con 0x7f1440068df0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 0x7f144012f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): picked mon.noname-a con 0x7f1440098570 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f144011bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): picked mon.noname-c con 0x7f14401323a0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1440131390 con 0x7f1440098570 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f144011f420 con 0x7f1440068df0 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:51.890 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f144011f700 con 0x7f14401323a0 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.878+0000 7f1445753700 10 monclient(hunting): authenticate will time out at 1468.175909s 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 0x7f144012f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f144011bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 0x7f144012f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46918/0 (socket says 172.21.15.6:46918) 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f144011bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:41332/0 (socket says 172.21.15.6:41332) 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 1 -- 172.21.15.6:0/3741035179 learned_addr learned my addr 172.21.15.6:0/3741035179 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): get_auth_request con 0x7f1440098570 auth_method 0 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.891 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): get_auth_request con 0x7f14401323a0 auth_method 0 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1440068df0 0x7f1440141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): get_auth_request con 0x7f1440068df0 auth_method 0 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:51.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient(hunting): handle_auth_done global_id 4601 payload 824 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 -- 172.21.15.6:0/3741035179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1440068df0 msgr2=0x7f1440141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1440068df0 0x7f1440141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 -- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 msgr2=0x7f144012f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 0x7f144012f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 monclient: found mon.noname-c 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:51.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:51.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143f7fe700 1 -- 172.21.15.6:0/3741035179 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14340094f0 con 0x7f14401323a0 2023-09-30T00:47:51.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.879+0000 7f143e7fc700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 0x7f144012f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f143f7fe700 1 -- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 msgr2=0x7f144011bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f143f7fe700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f144011bdc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1438009e50 tx=0x7f143800b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f143800ccb0 con 0x7f14401323a0 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: dump: 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:51.896 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: _finish_auth 0 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f143800f040 con 0x7f14401323a0 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 5 monclient: authenticate success, global_id 4601 2023-09-30T00:47:51.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 10 monclient: _renew_subs 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1430009420 con 0x7f14401323a0 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14380146d0 con 0x7f14401323a0 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1426ffd700 10 monclient: dump: 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:51.898 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 10 monclient: _renew_subs 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f143d7fa700 4 set_mon_vals no callback set 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.881+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1440141a70 con 0x7f14401323a0 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.882+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1438014a40 con 0x7f14401323a0 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.882+0000 7f1445753700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.882+0000 7f1445753700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:51.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.882+0000 7f1445753700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.882+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f144011faf0 con 0x7f14401323a0 2023-09-30T00:47:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.883+0000 7f1426ffd700 1 --2- 172.21.15.6:0/3741035179 >> v2:172.21.15.22:6812/142603 conn(0x7f142c0357a0 0x7f142c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:51.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.883+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1438048570 con 0x7f14401323a0 2023-09-30T00:47:51.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.883+0000 7f143effd700 1 --2- 172.21.15.6:0/3741035179 >> v2:172.21.15.22:6812/142603 conn(0x7f142c0357a0 0x7f142c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:51.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.883+0000 7f143effd700 10 monclient: get_auth_request con 0x7f142c0357a0 auth_method 0 2023-09-30T00:47:51.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.885+0000 7f143effd700 1 --2- 172.21.15.6:0/3741035179 >> v2:172.21.15.22:6812/142603 conn(0x7f142c0357a0 0x7f142c037c50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f14400a09d0 tx=0x7f1430017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:51.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.887+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f143800ce30 con 0x7f14401323a0 2023-09-30T00:47:51.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.887+0000 7f1426ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:51.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:51.887+0000 7f1426ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1445753700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1445753700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1445753700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f144011f8c0 con 0x7f14401323a0 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1426ffd700 1 -- 172.21.15.6:0/3741035179 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1438018020 con 0x7f14401323a0 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1426ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:52.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.138+0000 7f1426ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:52.139 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:52.139 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:52.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.140+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 >> v2:172.21.15.22:6812/142603 conn(0x7f142c0357a0 msgr2=0x7f142c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.140+0000 7f1445753700 1 --2- 172.21.15.6:0/3741035179 >> v2:172.21.15.22:6812/142603 conn(0x7f142c0357a0 0x7f142c037c50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f14400a09d0 tx=0x7f1430017000 comp rx=0 tx=0).stop 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.140+0000 7f1445753700 10 monclient: shutdown 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.140+0000 7f1445753700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.140+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 msgr2=0x7f144011bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.140+0000 7f1445753700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f144011bdc0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f1438009e50 tx=0x7f143800b040 comp rx=0 tx=0).stop 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 shutdown_connections 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 --2- 172.21.15.6:0/3741035179 >> v2:172.21.15.22:6812/142603 conn(0x7f142c0357a0 0x7f142c037c50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f14401323a0 0x7f144011bdc0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1440068df0 0x7f1440141530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 --2- 172.21.15.6:0/3741035179 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1440098570 0x7f144012f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:52.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 >> 172.21.15.6:0/3741035179 conn(0x7f14400a5480 msgr2=0x7f1440130ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 shutdown_connections 2023-09-30T00:47:52.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:52.141+0000 7f1445753700 1 -- 172.21.15.6:0/3741035179 wait complete. 2023-09-30T00:47:55.154 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:55.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.293+0000 7fde1bc21700 10 monclient: get_monmap_and_config 2023-09-30T00:47:55.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.293+0000 7fde1bc21700 10 monclient: build_initial_monmap 2023-09-30T00:47:55.293 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.293+0000 7fde1bc21700 10 monclient: monmap: 2023-09-30T00:47:55.293 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:55.294051+0000 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:55.294051+0000 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:55.294 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:55.295 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.294+0000 7fde1bc21700 1 Processor -- start 2023-09-30T00:47:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.294+0000 7fde1bc21700 1 -- start start 2023-09-30T00:47:55.298 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.294+0000 7fde1bc21700 10 monclient: init 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient: _add_conns ranks=[2,0,1,3,4] 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 0x7fde14132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient(hunting): picked mon.noname-b con 0x7fde14098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fde14098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fde14098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde140a1000 con 0x7fde14098a40 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1412fc00 con 0x7fde14098e90 2023-09-30T00:47:55.299 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde140a0a50 con 0x7fde14098670 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde1bc21700 10 monclient(hunting): authenticate will time out at 1471.592280s 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 0x7fde14132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde191bc700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fde14098a40 0x7fde1412ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:47154/0 (socket says 172.21.15.6:47154) 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde191bc700 1 -- 172.21.15.6:0/4204223559 learned_addr learned my addr 172.21.15.6:0/4204223559 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): get_auth_request con 0x7fde14098670 auth_method 0 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.300 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient(hunting): handle_auth_done global_id 4604 payload 274 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 1 -- 172.21.15.6:0/4204223559 >> v1:172.21.15.22:6789/0 conn(0x7fde14098e90 legacy=0x7fde1412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 1 -- 172.21.15.6:0/4204223559 >> v1:172.21.15.6:6789/0 conn(0x7fde14098a40 legacy=0x7fde1412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient: _finish_hunting 0 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 1 monclient: found mon.noname-b 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:47:55.301 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.295+0000 7fde199bd700 1 -- 172.21.15.6:0/4204223559 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde140a0730 con 0x7fde14098670 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.297+0000 7fde199bd700 1 -- 172.21.15.6:0/4204223559 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 msgr2=0x7fde14132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.297+0000 7fde199bd700 1 --2- 172.21.15.6:0/4204223559 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 0x7fde14132d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde10009910 tx=0x7fde100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2399162ca099d62d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 1 -- 172.21.15.6:0/4204223559 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde1000b9f0 con 0x7fde14098670 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: dump: 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.302 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: _finish_auth 0 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 1 -- 172.21.15.6:0/4204223559 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde1000bb50 con 0x7fde14098670 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 5 monclient: authenticate success, global_id 4604 2023-09-30T00:47:55.303 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 1 -- 172.21.15.6:0/4204223559 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde10011420 con 0x7fde14098670 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 4 set_mon_vals no callback set 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde189bb700 10 monclient: dump: 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:55.304 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 10 monclient: shutdown 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 -- 172.21.15.6:0/4204223559 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 msgr2=0x7fde14132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.305 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 --2- 172.21.15.6:0/4204223559 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 0x7fde14132d00 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fde10009910 tx=0x7fde100091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 -- 172.21.15.6:0/4204223559 shutdown_connections 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 --2- 172.21.15.6:0/4204223559 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098670 0x7fde14132d00 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 -- 172.21.15.6:0/4204223559 >> 172.21.15.6:0/4204223559 conn(0x7fde140a5480 msgr2=0x7fde140a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 -- 172.21.15.6:0/4204223559 shutdown_connections 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.298+0000 7fde1bc21700 1 -- 172.21.15.6:0/4204223559 wait complete. 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.299+0000 7fde1bc21700 10 monclient: build_initial_monmap 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.299+0000 7fde1bc21700 10 monclient: monmap: 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:55.306 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:55.307 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.299+0000 7fde1bc21700 1 Processor -- start 2023-09-30T00:47:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.299+0000 7fde1bc21700 1 -- start start 2023-09-30T00:47:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.299+0000 7fde1bc21700 10 monclient: init 2023-09-30T00:47:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:47:55.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 0x7fde14141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): picked mon.noname-a con 0x7fde14098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde14098a40 0x7fde1412c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): picked mon.noname-b con 0x7fde14098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 0x7fde1412c700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): picked mon.noname-c con 0x7fde14098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1411c5f0 con 0x7fde14098670 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1409f380 con 0x7fde14098a40 2023-09-30T00:47:55.312 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde140a3b70 con 0x7fde14098e90 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1bc21700 10 monclient(hunting): authenticate will time out at 1471.597269s 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 0x7fde14141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 0x7fde14141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46934/0 (socket says 172.21.15.6:46934) 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 1 -- 172.21.15.6:0/1168369523 learned_addr learned my addr 172.21.15.6:0/1168369523 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): get_auth_request con 0x7fde14098670 auth_method 0 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 0x7fde1412c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): get_auth_request con 0x7fde14098e90 auth_method 0 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:55.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde191bc700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde14098a40 0x7fde1412c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde191bc700 10 monclient(hunting): get_auth_request con 0x7fde14098a40 auth_method 0 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde191bc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde191bc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde191bc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient(hunting): handle_auth_done global_id 4607 payload 824 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 -- 172.21.15.6:0/1168369523 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde14098a40 msgr2=0x7fde1412c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde14098a40 0x7fde1412c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 -- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 msgr2=0x7fde14141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.315 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 0x7fde14141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient: _finish_hunting 0 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 monclient: found mon.noname-c 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde1a1be700 1 -- 172.21.15.6:0/1168369523 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde10009420 con 0x7fde14098e90 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.300+0000 7fde199bd700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 0x7fde14141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.302+0000 7fde1a1be700 1 -- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 msgr2=0x7fde1412c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.302+0000 7fde1a1be700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 0x7fde1412c700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde140a0650 tx=0x7fde0c002620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c012840 con 0x7fde14098e90 2023-09-30T00:47:55.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: dump: 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:55.317 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: _finish_auth 0 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 5 monclient: authenticate success, global_id 4607 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c0129c0 con 0x7fde14098e90 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 10 monclient: _renew_subs 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde140a1500 con 0x7fde14098e90 2023-09-30T00:47:55.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fde0c012d80 con 0x7fde14098e90 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde05ffb700 10 monclient: dump: 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:55.319 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde07fff700 4 set_mon_vals no callback set 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 10 monclient: _renew_subs 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.303+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde14141d80 con 0x7fde14098e90 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fde0c011590 con 0x7fde14098e90 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde05ffb700 1 --2- 172.21.15.6:0/1168369523 >> v2:172.21.15.22:6812/142603 conn(0x7fddf80357a0 0x7fddf8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:55.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fde0c047ba0 con 0x7fde14098e90 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde1bc21700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde1bc21700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde1bc21700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde199bd700 1 --2- 172.21.15.6:0/1168369523 >> v2:172.21.15.22:6812/142603 conn(0x7fddf80357a0 0x7fddf8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1411c950 con 0x7fde14098e90 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.304+0000 7fde199bd700 10 monclient: get_auth_request con 0x7fddf80357a0 auth_method 0 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.307+0000 7fde199bd700 1 --2- 172.21.15.6:0/1168369523 >> v2:172.21.15.22:6812/142603 conn(0x7fddf80357a0 0x7fddf8037c50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde10006dc0 tx=0x7fde10016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.309+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fde1411c950 con 0x7fde14098e90 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.309+0000 7fde05ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:55.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.309+0000 7fde05ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.556+0000 7fde1bc21700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.556+0000 7fde1bc21700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:55.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.556+0000 7fde1bc21700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:47:55.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.556+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fde1411c950 con 0x7fde14098e90 2023-09-30T00:47:55.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.557+0000 7fde05ffb700 1 -- 172.21.15.6:0/1168369523 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fde1411c950 con 0x7fde14098e90 2023-09-30T00:47:55.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.557+0000 7fde05ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:55.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.557+0000 7fde05ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:55.557 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:55.557 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 >> v2:172.21.15.22:6812/142603 conn(0x7fddf80357a0 msgr2=0x7fddf8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 --2- 172.21.15.6:0/1168369523 >> v2:172.21.15.22:6812/142603 conn(0x7fddf80357a0 0x7fddf8037c50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde10006dc0 tx=0x7fde10016000 comp rx=0 tx=0).stop 2023-09-30T00:47:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 10 monclient: shutdown 2023-09-30T00:47:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 msgr2=0x7fde1412c700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:55.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 0x7fde1412c700 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fde140a0650 tx=0x7fde0c002620 comp rx=0 tx=0).stop 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 shutdown_connections 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 --2- 172.21.15.6:0/1168369523 >> v2:172.21.15.22:6812/142603 conn(0x7fddf80357a0 0x7fddf8037c50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fde14098e90 0x7fde1412c700 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fde14098670 0x7fde14141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 --2- 172.21.15.6:0/1168369523 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fde14098a40 0x7fde1412c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 >> 172.21.15.6:0/1168369523 conn(0x7fde140a5480 msgr2=0x7fde1412fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 shutdown_connections 2023-09-30T00:47:55.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:55.559+0000 7fde1bc21700 1 -- 172.21.15.6:0/1168369523 wait complete. 2023-09-30T00:47:58.573 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:47:58.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.711+0000 7f2d1f444700 10 monclient: get_monmap_and_config 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.711+0000 7f2d1f444700 10 monclient: build_initial_monmap 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.711+0000 7f2d1f444700 10 monclient: monmap: 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:47:58.712131+0000 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:47:58.712131+0000 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:47:58.712 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:47:58.713 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:47:58.713 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:47:58.713 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:47:58.713 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:58.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.712+0000 7f2d1f444700 1 Processor -- start 2023-09-30T00:47:58.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.712+0000 7f2d1f444700 1 -- start start 2023-09-30T00:47:58.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.712+0000 7f2d1f444700 10 monclient: init 2023-09-30T00:47:58.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:58.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient: _add_conns ranks=[4,1,0,2,3] 2023-09-30T00:47:58.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f2d18098670 0x7f2d18132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient(hunting): picked mon.noname-c con 0x7f2d18098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2d18098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2d18098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d180a1000 con 0x7f2d18098e90 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d1812fc00 con 0x7f2d18098a40 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d180a0a50 con 0x7f2d18098670 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1f444700 10 monclient(hunting): authenticate will time out at 1475.010349s 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d9e1700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2d18098e90 0x7f2d1812f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:42420/0 (socket says 172.21.15.6:42420) 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d9e1700 1 -- 172.21.15.6:0/2040861060 learned_addr learned my addr 172.21.15.6:0/2040861060 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:47:58.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/2040861060 >> v2:172.21.15.22:3300/0 conn(0x7f2d18098670 0x7f2d18132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): get_auth_request con 0x7f2d18098670 auth_method 0 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.713+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_done global_id 4609 payload 274 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 1 -- 172.21.15.6:0/2040861060 >> v1:172.21.15.22:6789/0 conn(0x7f2d18098a40 legacy=0x7f2d1812ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 1 -- 172.21.15.6:0/2040861060 >> v1:172.21.15.6:6789/0 conn(0x7f2d18098e90 legacy=0x7f2d1812f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:47:58.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 10 monclient: _finish_hunting 0 2023-09-30T00:47:58.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 1 monclient: found mon.noname-c 2023-09-30T00:47:58.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:58.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:47:58.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.714+0000 7f2d1d1e0700 1 -- 172.21.15.6:0/2040861060 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d180a0730 con 0x7f2d18098670 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d1d1e0700 1 -- 172.21.15.6:0/2040861060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098670 msgr2=0x7f2d18132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/2040861060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098670 0x7f2d18132d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d140098e0 tx=0x7f2d140091a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a79f13e9db2ef71f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 1 -- 172.21.15.6:0/2040861060 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d1400b9f0 con 0x7f2d18098670 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: dump: 2023-09-30T00:47:58.724 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: _finish_auth 0 2023-09-30T00:47:58.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 1 -- 172.21.15.6:0/2040861060 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d1400bb50 con 0x7f2d18098670 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d1f444700 5 monclient: authenticate success, global_id 4609 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d1f444700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d1f444700 10 monclient: get_monmap_and_config success 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 1 -- 172.21.15.6:0/2040861060 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d14011420 con 0x7f2d18098670 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d1f444700 4 set_mon_vals no callback set 2023-09-30T00:47:58.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.716+0000 7f2d0ffff700 10 monclient: dump: 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:58.727 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 10 monclient: shutdown 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 -- 172.21.15.6:0/2040861060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098670 msgr2=0x7f2d18132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 --2- 172.21.15.6:0/2040861060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098670 0x7f2d18132d00 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f2d140098e0 tx=0x7f2d140091a0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 -- 172.21.15.6:0/2040861060 shutdown_connections 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 --2- 172.21.15.6:0/2040861060 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098670 0x7f2d18132d00 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 -- 172.21.15.6:0/2040861060 >> 172.21.15.6:0/2040861060 conn(0x7f2d180a5480 msgr2=0x7f2d180a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 -- 172.21.15.6:0/2040861060 shutdown_connections 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.717+0000 7f2d1f444700 1 -- 172.21.15.6:0/2040861060 wait complete. 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.718+0000 7f2d1f444700 10 monclient: build_initial_monmap 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.718+0000 7f2d1f444700 10 monclient: monmap: 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:58.729 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.718+0000 7f2d1f444700 1 Processor -- start 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.718+0000 7f2d1f444700 1 -- start start 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.718+0000 7f2d1f444700 10 monclient: init 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient: _reopen_session rank -1 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 0x7f2d181417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): picked mon.noname-c con 0x7f2d18098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 0x7f2d1812c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): picked mon.noname-a con 0x7f2d18098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098e90 0x7f2d1809ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): picked mon.noname-b con 0x7f2d18098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d181426f0 con 0x7f2d18098a40 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1809f230 con 0x7f2d18098e90 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): start opening mon connection 2023-09-30T00:47:58.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d1812ed70 con 0x7f2d18098670 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): _renew_subs 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1f444700 10 monclient(hunting): authenticate will time out at 1475.016220s 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 0x7f2d1812c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 0x7f2d181417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 0x7f2d1812c150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58494/0 (socket says 172.21.15.6:58494) 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 0x7f2d181417d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36306/0 (socket says 172.21.15.6:36306) 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 -- 172.21.15.6:0/3892822774 learned_addr learned my addr 172.21.15.6:0/3892822774 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): get_auth_request con 0x7f2d18098a40 auth_method 0 2023-09-30T00:47:58.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): get_auth_request con 0x7f2d18098670 auth_method 0 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:47:58.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d9e1700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098e90 0x7f2d1809ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d9e1700 10 monclient(hunting): get_auth_request con 0x7f2d18098e90 auth_method 0 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d9e1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d9e1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d9e1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient(hunting): handle_auth_done global_id 4599 payload 824 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 -- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 msgr2=0x7f2d181417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 0x7f2d181417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 -- 172.21.15.6:0/3892822774 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098e90 msgr2=0x7f2d1809ca80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098e90 0x7f2d1809ca80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient: _finish_hunting 0 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 monclient: found mon.noname-a 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1c9df700 1 -- 172.21.15.6:0/3892822774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d14009420 con 0x7f2d18098a40 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.719+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 0x7f2d181417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.721+0000 7f2d1c9df700 1 -- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 msgr2=0x7f2d1812c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.721+0000 7f2d1c9df700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 0x7f2d1812c150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800c5d0 tx=0x7f2d08007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d080117a0 con 0x7f2d18098a40 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: dump: 2023-09-30T00:47:58.735 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: _finish_auth 0 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:47:58.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 5 monclient: authenticate success, global_id 4599 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 10 monclient: _renew_subs 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d1811c2a0 con 0x7f2d18098a40 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d08011900 con 0x7f2d18098a40 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d08011cc0 con 0x7f2d18098a40 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0d7fa700 10 monclient: dump: 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:47:58.737 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d0f7fe700 4 set_mon_vals no callback set 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 10 monclient: _renew_subs 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.722+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d18141d10 con 0x7f2d18098a40 2023-09-30T00:47:58.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f2d08012630 con 0x7f2d18098a40 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1f444700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1f444700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1f444700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d18072b00 con 0x7f2d18098a40 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d0d7fa700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f2d08047630 con 0x7f2d18098a40 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1d1e0700 10 monclient: get_auth_request con 0x7f2cfc0357a0 auth_method 0 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1d1e0700 1 -- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 msgr2=0x7f2cfc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1d1e0700 1 -- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 msgr2=0x7f2cfc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 secure :-1 s=COMPRESSION_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0x7f2d140030c0 tx=0x7f2d14016000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2023-09-30T00:47:58.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.723+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2023-09-30T00:47:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.728+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f2d080108b0 con 0x7f2d18098a40 2023-09-30T00:47:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.728+0000 7f2d0d7fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:47:58.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.728+0000 7f2d0d7fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:47:58.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.924+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:47:58.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.924+0000 7f2d1d1e0700 10 monclient: get_auth_request con 0x7f2cfc0357a0 auth_method 0 2023-09-30T00:47:58.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.927+0000 7f2d1d1e0700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d140093a0 tx=0x7f2d14021000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:47:58.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.972+0000 7f2d1f444700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:47:58.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.972+0000 7f2d1f444700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:58.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.972+0000 7f2d1f444700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:47:58.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.972+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2d18063920 con 0x7f2d18098a40 2023-09-30T00:47:58.972 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.973+0000 7f2d0d7fa700 1 -- 172.21.15.6:0/3892822774 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f2d080102e0 con 0x7f2d18098a40 2023-09-30T00:47:58.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.973+0000 7f2d0d7fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:47:58.973 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.973+0000 7f2d0d7fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:47:58.973 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:47:58.973 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 msgr2=0x7f2cfc037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f2d140093a0 tx=0x7f2d14021000 comp rx=0 tx=0).stop 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 10 monclient: shutdown 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 msgr2=0x7f2d1812c150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 0x7f2d1812c150 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2d0800c5d0 tx=0x7f2d08007f60 comp rx=0 tx=0).stop 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 shutdown_connections 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 --2- 172.21.15.6:0/3892822774 >> v2:172.21.15.22:6812/142603 conn(0x7f2cfc0357a0 0x7f2cfc037c50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2d18098e90 0x7f2d1809ca80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2d18098670 0x7f2d181417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.975 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 --2- 172.21.15.6:0/3892822774 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2d18098a40 0x7f2d1812c150 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:47:58.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 >> 172.21.15.6:0/3892822774 conn(0x7f2d180a5480 msgr2=0x7f2d1812f450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:47:58.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 shutdown_connections 2023-09-30T00:47:58.976 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:47:58.975+0000 7f2d1f444700 1 -- 172.21.15.6:0/3892822774 wait complete. 2023-09-30T00:48:01.989 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.127+0000 7f20e751b700 10 monclient: get_monmap_and_config 2023-09-30T00:48:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.127+0000 7f20e751b700 10 monclient: build_initial_monmap 2023-09-30T00:48:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.127+0000 7f20e751b700 10 monclient: monmap: 2023-09-30T00:48:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:02.127 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:02.127879+0000 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:02.127879+0000 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:02.128 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 1 Processor -- start 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 1 -- start start 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 10 monclient: init 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 10 monclient: _add_conns ranks=[2,3,4,1,0] 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 0x7f20e0132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 10 monclient(hunting): picked mon.noname-b con 0x7f20e0098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f20e0098a40 0x7f20e012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 10 monclient(hunting): picked mon.noname-a con 0x7f20e0098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f20e0098e90 0x7f20e012f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.128+0000 7f20e751b700 10 monclient(hunting): picked mon.noname-c con 0x7f20e0098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e00a0a50 con 0x7f20e0098a40 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e00a1000 con 0x7f20e0098e90 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00a0730 con 0x7f20e0098670 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e751b700 10 monclient(hunting): authenticate will time out at 1478.426022s 2023-09-30T00:48:02.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 0x7f20e0132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f20e0098a40 0x7f20e012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 0x7f20e0132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36310/0 (socket says 172.21.15.6:36310) 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f20e0098a40 0x7f20e012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58510/0 (socket says 172.21.15.6:58510) 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 1 -- 172.21.15.6:0/3871961563 learned_addr learned my addr 172.21.15.6:0/3871961563 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): get_auth_request con 0x7f20e0098670 auth_method 0 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): get_auth_request con 0x7f20e0098a40 auth_method 0 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:02.135 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e5ab8700 1 --2- 172.21.15.6:0/3871961563 >> v2:172.21.15.22:3300/0 conn(0x7f20e0098e90 0x7f20e012f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_done global_id 4602 payload 274 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 -- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 msgr2=0x7f20e0132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 0x7f20e0132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 -- 172.21.15.6:0/3871961563 >> v2:172.21.15.22:3300/0 conn(0x7f20e0098e90 msgr2=0x7f20e012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:02.136 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/3871961563 >> v2:172.21.15.22:3300/0 conn(0x7f20e0098e90 0x7f20e012f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient: _finish_hunting 0 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 monclient: found mon.noname-a 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e4ab6700 1 -- 172.21.15.6:0/3871961563 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e0068df0 con 0x7f20e0098a40 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e5ab8700 10 monclient: get_auth_request con 0x7f20e0098e90 auth_method 0 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e5ab8700 1 --2- 172.21.15.6:0/3871961563 >> v2:172.21.15.22:3300/0 conn(0x7f20e0098e90 0x7f20e012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.129+0000 7f20e52b7700 1 --2- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 0x7f20e0132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.131+0000 7f20e4ab6700 1 -- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098a40 msgr2=0x7f20e012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.131+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098a40 0x7f20e012ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20e00a1df0 tx=0x7f20d00091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=756a38ea4bd633f8 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 1 -- 172.21.15.6:0/3871961563 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0011840 con 0x7f20e0098a40 2023-09-30T00:48:02.137 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: dump: 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:02.138 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: _finish_auth 0 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 5 monclient: authenticate success, global_id 4602 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 1 -- 172.21.15.6:0/3871961563 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d00119a0 con 0x7f20e0098a40 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 1 -- 172.21.15.6:0/3871961563 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20d0011d40 con 0x7f20e0098a40 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 4 set_mon_vals no callback set 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:02.139 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20d7fff700 10 monclient: dump: 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 10 monclient: shutdown 2023-09-30T00:48:02.140 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 -- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098a40 msgr2=0x7f20e012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 --2- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098a40 0x7f20e012ede0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f20e00a1df0 tx=0x7f20d00091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 -- 172.21.15.6:0/3871961563 shutdown_connections 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 --2- 172.21.15.6:0/3871961563 >> v2:172.21.15.22:3300/0 conn(0x7f20e0098e90 0x7f20e012f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 --2- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098670 0x7f20e0132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 --2- 172.21.15.6:0/3871961563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098a40 0x7f20e012ede0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 -- 172.21.15.6:0/3871961563 >> 172.21.15.6:0/3871961563 conn(0x7f20e00a5480 msgr2=0x7f20e00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 -- 172.21.15.6:0/3871961563 shutdown_connections 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.132+0000 7f20e751b700 1 -- 172.21.15.6:0/3871961563 wait complete. 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.133+0000 7f20e751b700 10 monclient: build_initial_monmap 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.133+0000 7f20e751b700 10 monclient: monmap: 2023-09-30T00:48:02.141 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.133+0000 7f20e751b700 1 Processor -- start 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.133+0000 7f20e751b700 1 -- start start 2023-09-30T00:48:02.142 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.133+0000 7f20e751b700 10 monclient: init 2023-09-30T00:48:02.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:02.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:48:02.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 0x7f20e012bc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.146 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): picked mon.noname-a con 0x7f20e0098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098a40 0x7f20e012c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): picked mon.noname-c con 0x7f20e0098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20e0098e90 0x7f20e012c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): picked mon.noname-b con 0x7f20e0098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e009b920 con 0x7f20e0098670 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e00a1530 con 0x7f20e0098e90 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e009abd0 con 0x7f20e0098a40 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:02.147 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e751b700 10 monclient(hunting): authenticate will time out at 1478.431529s 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 0x7f20e012bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 0x7f20e012bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58516/0 (socket says 172.21.15.6:58516) 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 -- 172.21.15.6:0/2679446045 learned_addr learned my addr 172.21.15.6:0/2679446045 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): get_auth_request con 0x7f20e0098670 auth_method 0 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098a40 0x7f20e012c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): get_auth_request con 0x7f20e0098a40 auth_method 0 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:02.148 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e5ab8700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20e0098e90 0x7f20e012c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient(hunting): handle_auth_done global_id 4605 payload 824 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 -- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098a40 msgr2=0x7f20e012c180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098a40 0x7f20e012c180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 -- 172.21.15.6:0/2679446045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20e0098e90 msgr2=0x7f20e012c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.149 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20e0098e90 0x7f20e012c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient: _finish_hunting 0 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 monclient: found mon.noname-a 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e52b7700 1 -- 172.21.15.6:0/2679446045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20dc0094f0 con 0x7f20e0098670 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e5ab8700 10 monclient: get_auth_request con 0x7f20e0098e90 auth_method 0 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e5ab8700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20e0098e90 0x7f20e012c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.134+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098a40 0x7f20e012c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e52b7700 1 -- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 msgr2=0x7f20e012bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e52b7700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 0x7f20e012bc40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00ec50 tx=0x7f20dc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0097d0 con 0x7f20e0098670 2023-09-30T00:48:02.150 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: dump: 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:02.151 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: _finish_auth 0 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 5 monclient: authenticate success, global_id 4605 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20dc00cb30 con 0x7f20e0098670 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 10 monclient: _renew_subs 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20d0009420 con 0x7f20e0098670 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f20dc013470 con 0x7f20e0098670 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:02.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d57fa700 10 monclient: dump: 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20d77fe700 4 set_mon_vals no callback set 2023-09-30T00:48:02.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 10 monclient: _renew_subs 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.137+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e009cdf0 con 0x7f20e0098670 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f20dc013820 con 0x7f20e0098670 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20d57fa700 1 --2- 172.21.15.6:0/2679446045 >> v2:172.21.15.22:6812/142603 conn(0x7f20c40357f0 0x7f20c4037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f20dc048020 con 0x7f20e0098670 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20e751b700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20e751b700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20e751b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e009d1b0 con 0x7f20e0098670 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/2679446045 >> v2:172.21.15.22:6812/142603 conn(0x7f20c40357f0 0x7f20c4037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:02.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.138+0000 7f20e4ab6700 10 monclient: get_auth_request con 0x7f20c40357f0 auth_method 0 2023-09-30T00:48:02.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.141+0000 7f20e4ab6700 1 --2- 172.21.15.6:0/2679446045 >> v2:172.21.15.22:6812/142603 conn(0x7f20c40357f0 0x7f20c4037ca0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f20e012fdc0 tx=0x7f20d0017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:02.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.143+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f20dc01b9a0 con 0x7f20e0098670 2023-09-30T00:48:02.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.143+0000 7f20d57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:02.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.143+0000 7f20d57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20e751b700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20e751b700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20e751b700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f20e0072b00 con 0x7f20e0098670 2023-09-30T00:48:02.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20d57fa700 1 -- 172.21.15.6:0/2679446045 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f20dc011350 con 0x7f20e0098670 2023-09-30T00:48:02.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20d57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:02.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.390+0000 7f20d57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:02.391 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:02.391 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:02.392 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 >> v2:172.21.15.22:6812/142603 conn(0x7f20c40357f0 msgr2=0x7f20c4037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 --2- 172.21.15.6:0/2679446045 >> v2:172.21.15.22:6812/142603 conn(0x7f20c40357f0 0x7f20c4037ca0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f20e012fdc0 tx=0x7f20d0017000 comp rx=0 tx=0).stop 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 10 monclient: shutdown 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 msgr2=0x7f20e012bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 0x7f20e012bc40 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00ec50 tx=0x7f20dc009250 comp rx=0 tx=0).stop 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 shutdown_connections 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 --2- 172.21.15.6:0/2679446045 >> v2:172.21.15.22:6812/142603 conn(0x7f20c40357f0 0x7f20c4037ca0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f20e0098e90 0x7f20e012c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f20e0098670 0x7f20e012bc40 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 --2- 172.21.15.6:0/2679446045 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f20e0098a40 0x7f20e012c180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:02.393 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 >> 172.21.15.6:0/2679446045 conn(0x7f20e00a5480 msgr2=0x7f20e00975f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:02.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 shutdown_connections 2023-09-30T00:48:02.394 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:02.393+0000 7f20e751b700 1 -- 172.21.15.6:0/2679446045 wait complete. 2023-09-30T00:48:05.406 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.544+0000 7f1ebb226700 10 monclient: get_monmap_and_config 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.544+0000 7f1ebb226700 10 monclient: build_initial_monmap 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.544+0000 7f1ebb226700 10 monclient: monmap: 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:05.545593+0000 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:05.545593+0000 2023-09-30T00:48:05.545 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:05.546 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 Processor -- start 2023-09-30T00:48:05.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 -- start start 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient: init 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1eb4098670 0x7f1eb4132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): picked mon.noname-a con 0x7f1eb4098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1eb4098a40 0x7f1eb412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): picked mon.noname-c con 0x7f1eb4098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb412f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): picked mon.noname-b con 0x7f1eb4098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40a0a50 con 0x7f1eb4098670 2023-09-30T00:48:05.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1eb40a1000 con 0x7f1eb4098a40 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40a0730 con 0x7f1eb4098e90 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1ebb226700 10 monclient(hunting): authenticate will time out at 1481.843842s 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1eb8fc2700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f1eb4098670 0x7f1eb4132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.545+0000 7f1eb97c3700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb412f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb412f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36324/0 (socket says 172.21.15.6:36324) 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 -- 172.21.15.6:0/2068940128 learned_addr learned my addr 172.21.15.6:0/2068940128 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): get_auth_request con 0x7f1eb4098670 auth_method 0 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:05.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): get_auth_request con 0x7f1eb4098e90 auth_method 0 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb3fff700 1 --2- 172.21.15.6:0/2068940128 >> v2:172.21.15.22:3300/0 conn(0x7f1eb4098a40 0x7f1eb412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb3fff700 10 monclient(hunting): get_auth_request con 0x7f1eb4098a40 auth_method 0 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_done global_id 4619 payload 274 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 -- 172.21.15.6:0/2068940128 >> v2:172.21.15.22:3300/0 conn(0x7f1eb4098a40 msgr2=0x7f1eb412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 --2- 172.21.15.6:0/2068940128 >> v2:172.21.15.22:3300/0 conn(0x7f1eb4098a40 0x7f1eb412ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 -- 172.21.15.6:0/2068940128 >> v2:172.21.15.6:3300/0 conn(0x7f1eb4098670 msgr2=0x7f1eb4132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 --2- 172.21.15.6:0/2068940128 >> v2:172.21.15.6:3300/0 conn(0x7f1eb4098670 0x7f1eb4132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient: _finish_hunting 0 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 monclient: found mon.noname-b 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:05.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb97c3700 1 -- 172.21.15.6:0/2068940128 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1eb4068df0 con 0x7f1eb4098e90 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.546+0000 7f1eb8fc2700 1 --2- 172.21.15.6:0/2068940128 >> v2:172.21.15.6:3300/0 conn(0x7f1eb4098670 0x7f1eb4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb97c3700 1 -- 172.21.15.6:0/2068940128 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 msgr2=0x7f1eb412f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb97c3700 1 --2- 172.21.15.6:0/2068940128 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb412f320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4063790 tx=0x7f1eac009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=203c37d821ffe7f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 1 -- 172.21.15.6:0/2068940128 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac00e070 con 0x7f1eb4098e90 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: dump: 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.555 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 1 -- 172.21.15.6:0/2068940128 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1eac00bc30 con 0x7f1eb4098e90 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1ebb226700 5 monclient: authenticate success, global_id 4619 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1ebb226700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:05.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1ebb226700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1ebb226700 4 set_mon_vals no callback set 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 1 -- 172.21.15.6:0/2068940128 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac0092e0 con 0x7f1eb4098e90 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.548+0000 7f1eb37fe700 10 monclient: dump: 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:05.557 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 10 monclient: shutdown 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 -- 172.21.15.6:0/2068940128 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 msgr2=0x7f1eb412f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 --2- 172.21.15.6:0/2068940128 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb412f320 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4063790 tx=0x7f1eac009840 comp rx=0 tx=0).stop 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 -- 172.21.15.6:0/2068940128 shutdown_connections 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 --2- 172.21.15.6:0/2068940128 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb412f320 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 --2- 172.21.15.6:0/2068940128 >> v2:172.21.15.6:3300/0 conn(0x7f1eb4098670 0x7f1eb4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 --2- 172.21.15.6:0/2068940128 >> v2:172.21.15.22:3300/0 conn(0x7f1eb4098a40 0x7f1eb412ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 -- 172.21.15.6:0/2068940128 >> 172.21.15.6:0/2068940128 conn(0x7f1eb40a5480 msgr2=0x7f1eb40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 -- 172.21.15.6:0/2068940128 shutdown_connections 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 1 -- 172.21.15.6:0/2068940128 wait complete. 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.549+0000 7f1ebb226700 10 monclient: build_initial_monmap 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.550+0000 7f1ebb226700 10 monclient: monmap: 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:05.559 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:05.560 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:05.560 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:05.560 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.550+0000 7f1ebb226700 1 Processor -- start 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.550+0000 7f1ebb226700 1 -- start start 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.550+0000 7f1ebb226700 10 monclient: init 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): picked mon.noname-a con 0x7f1eb4098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1eb4098a40 0x7f1eb412c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): picked mon.noname-b con 0x7f1eb4098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb409cb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): picked mon.noname-c con 0x7f1eb4098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40a3c10 con 0x7f1eb4098670 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb40a1500 con 0x7f1eb4098a40 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1eb409e830 con 0x7f1eb4098e90 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1ebb226700 10 monclient(hunting): authenticate will time out at 1481.849259s 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:58546/0 (socket says 172.21.15.6:58546) 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 -- 172.21.15.6:0/3239398870 learned_addr learned my addr 172.21.15.6:0/3239398870 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): get_auth_request con 0x7f1eb4098670 auth_method 0 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb409cb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): get_auth_request con 0x7f1eb4098e90 auth_method 0 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:05.566 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb3fff700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1eb4098a40 0x7f1eb412c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb3fff700 10 monclient(hunting): get_auth_request con 0x7f1eb4098a40 auth_method 0 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient(hunting): handle_auth_done global_id 4611 payload 824 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 -- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 msgr2=0x7f1eb409cb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:05.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb409cb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 -- 172.21.15.6:0/3239398870 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1eb4098a40 msgr2=0x7f1eb412c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1eb4098a40 0x7f1eb412c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient: _finish_hunting 0 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 monclient: found mon.noname-a 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb8fc2700 1 -- 172.21.15.6:0/3239398870 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea4009420 con 0x7f1eb4098670 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.551+0000 7f1eb97c3700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb409cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb8fc2700 1 -- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 msgr2=0x7f1eb412be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb8fc2700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400ec80 tx=0x7f1ea40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea400cb00 con 0x7f1eb4098670 2023-09-30T00:48:05.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: dump: 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:05.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: _finish_auth 0 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ea400cc60 con 0x7f1eb4098670 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 5 monclient: authenticate success, global_id 4611 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 10 monclient: _renew_subs 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1eac009ae0 con 0x7f1eb4098670 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ea4013470 con 0x7f1eb4098670 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb0ff9700 10 monclient: dump: 2023-09-30T00:48:05.570 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 10 monclient: _renew_subs 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1eb2ffd700 4 set_mon_vals no callback set 2023-09-30T00:48:05.571 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.554+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1eb412d6b0 con 0x7f1eb4098670 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1eb8fc2700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400ec80 tx=0x7f1ea40091a0 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f1ea4047d00 osd_map(17..17 src has 1..17) v4 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1ea40137e0 con 0x7f1eb4098670 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1ebb226700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1ebb226700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1ebb226700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1eb412dd90 con 0x7f1eb4098670 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1eb0ff9700 1 --2- 172.21.15.6:0/3239398870 >> v2:172.21.15.22:6812/142603 conn(0x7f1ea00357f0 0x7f1ea0037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1eb3fff700 1 --2- 172.21.15.6:0/3239398870 >> v2:172.21.15.22:6812/142603 conn(0x7f1ea00357f0 0x7f1ea0037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.555+0000 7f1eb3fff700 10 monclient: get_auth_request con 0x7f1ea00357f0 auth_method 0 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.558+0000 7f1eb3fff700 1 --2- 172.21.15.6:0/3239398870 >> v2:172.21.15.22:6812/142603 conn(0x7f1ea00357f0 0x7f1ea0037ca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8006fd0 tx=0x7f1ea8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:05.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:05.560+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1ea4047d00 con 0x7f1eb4098670 2023-09-30T00:48:06.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.383+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1ea40238e0 con 0x7f1eb4098670 2023-09-30T00:48:06.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.383+0000 7f1eb0ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:06.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.383+0000 7f1eb0ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1ebb226700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1ebb226700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1ebb226700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1eb412dab0 con 0x7f1eb4098670 2023-09-30T00:48:06.630 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1eb0ff9700 1 -- 172.21.15.6:0/3239398870 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1ea40232f0 con 0x7f1eb4098670 2023-09-30T00:48:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1eb0ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:06.631 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.629+0000 7f1eb0ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:06.631 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:06.631 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:06.632 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.631+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 >> v2:172.21.15.22:6812/142603 conn(0x7f1ea00357f0 msgr2=0x7f1ea0037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.631+0000 7f1ebb226700 1 --2- 172.21.15.6:0/3239398870 >> v2:172.21.15.22:6812/142603 conn(0x7f1ea00357f0 0x7f1ea0037ca0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1ea8006fd0 tx=0x7f1ea8008040 comp rx=0 tx=0).stop 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.631+0000 7f1ebb226700 10 monclient: shutdown 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 msgr2=0x7f1eb412be00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f1ea400ec80 tx=0x7f1ea40091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 shutdown_connections 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 --2- 172.21.15.6:0/3239398870 >> v2:172.21.15.22:6812/142603 conn(0x7f1ea00357f0 0x7f1ea0037ca0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1eb4098e90 0x7f1eb409cb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1eb4098670 0x7f1eb412be00 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 --2- 172.21.15.6:0/3239398870 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1eb4098a40 0x7f1eb412c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:06.633 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 >> 172.21.15.6:0/3239398870 conn(0x7f1eb40a5480 msgr2=0x7f1eb4069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 shutdown_connections 2023-09-30T00:48:06.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:06.632+0000 7f1ebb226700 1 -- 172.21.15.6:0/3239398870 wait complete. 2023-09-30T00:48:09.647 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.786+0000 7fc11c053700 10 monclient: get_monmap_and_config 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.786+0000 7fc11c053700 10 monclient: build_initial_monmap 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.786+0000 7fc11c053700 10 monclient: monmap: 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:09.787634+0000 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:09.787634+0000 2023-09-30T00:48:09.787 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:09.788 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:09.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 1 Processor -- start 2023-09-30T00:48:09.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 1 -- start start 2023-09-30T00:48:09.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient: init 2023-09-30T00:48:09.790 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient: _add_conns ranks=[0,1,4,2,3] 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fc114098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fc114132d00 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc114138d40 0x7fc11413b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient(hunting): picked mon.noname-c con 0x7fc114138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140a1000 con 0x7fc114098670 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1140a0730 con 0x7fc114132d00 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc114099150 con 0x7fc114138d40 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:09.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.787+0000 7fc11c053700 10 monclient(hunting): authenticate will time out at 1486.085909s 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc119def700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fc114098670 0x7fc114098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51094/0 (socket says 172.21.15.6:51094) 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc119def700 1 -- 172.21.15.6:0/1833982198 learned_addr learned my addr 172.21.15.6:0/1833982198 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 1 --2- 172.21.15.6:0/1833982198 >> v2:172.21.15.22:3300/0 conn(0x7fc114138d40 0x7fc11413b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): get_auth_request con 0x7fc114138d40 auth_method 0 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:09.792 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_done global_id 4621 payload 274 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 1 -- 172.21.15.6:0/1833982198 >> v1:172.21.15.22:6789/0 conn(0x7fc114132d00 legacy=0x7fc114137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 1 -- 172.21.15.6:0/1833982198 >> v1:172.21.15.6:6789/0 conn(0x7fc114098670 legacy=0x7fc114098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient: _finish_hunting 0 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 1 monclient: found mon.noname-c 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:48:09.793 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.788+0000 7fc11a5f0700 1 -- 172.21.15.6:0/1833982198 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc114068df0 con 0x7fc114138d40 2023-09-30T00:48:09.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.790+0000 7fc11a5f0700 1 -- 172.21.15.6:0/1833982198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114138d40 msgr2=0x7fc11413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:09.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.790+0000 7fc11a5f0700 1 --2- 172.21.15.6:0/1833982198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114138d40 0x7fc11413b120 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009e60 tx=0x7fc10c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b14c8fb8edaff988 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:09.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 1 -- 172.21.15.6:0/1833982198 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c00e070 con 0x7fc114138d40 2023-09-30T00:48:09.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:09.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: dump: 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:09.798 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: _finish_auth 0 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 5 monclient: authenticate success, global_id 4621 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 1 -- 172.21.15.6:0/1833982198 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc10c012430 con 0x7fc114138d40 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 4 set_mon_vals no callback set 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 1 -- 172.21.15.6:0/1833982198 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc10c0127d0 con 0x7fc114138d40 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:09.799 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc118ded700 10 monclient: dump: 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:09.800 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 10 monclient: shutdown 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 1 -- 172.21.15.6:0/1833982198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114138d40 msgr2=0x7fc11413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 1 --2- 172.21.15.6:0/1833982198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114138d40 0x7fc11413b120 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc10c009e60 tx=0x7fc10c009780 comp rx=0 tx=0).stop 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 1 -- 172.21.15.6:0/1833982198 shutdown_connections 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 1 --2- 172.21.15.6:0/1833982198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114138d40 0x7fc11413b120 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.791+0000 7fc11c053700 1 -- 172.21.15.6:0/1833982198 >> 172.21.15.6:0/1833982198 conn(0x7fc1140a5480 msgr2=0x7fc1140a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 1 -- 172.21.15.6:0/1833982198 shutdown_connections 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 1 -- 172.21.15.6:0/1833982198 wait complete. 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 10 monclient: build_initial_monmap 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 10 monclient: monmap: 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:09.801 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 1 Processor -- start 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 1 -- start start 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.792+0000 7fc11c053700 10 monclient: init 2023-09-30T00:48:09.802 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 0x7fc114141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): picked mon.noname-a con 0x7fc114098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114132d00 0x7fc114138330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): picked mon.noname-b con 0x7fc114132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc114138d40 0x7fc11413f500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): picked mon.noname-c con 0x7fc114138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc114133af0 con 0x7fc114098670 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:09.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc11412f1e0 con 0x7fc114132d00 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc114133530 con 0x7fc114138d40 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11c053700 10 monclient(hunting): authenticate will time out at 1486.091374s 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 0x7fc114141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 0x7fc114141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:49660/0 (socket says 172.21.15.6:49660) 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 -- 172.21.15.6:0/2905393464 learned_addr learned my addr 172.21.15.6:0/2905393464 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): get_auth_request con 0x7fc114098670 auth_method 0 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:09.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc114138d40 0x7fc11413f500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): get_auth_request con 0x7fc114138d40 auth_method 0 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc1195ee700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114132d00 0x7fc114138330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:09.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc1195ee700 10 monclient(hunting): get_auth_request con 0x7fc114132d00 auth_method 0 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc1195ee700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc1195ee700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc1195ee700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient(hunting): handle_auth_done global_id 4614 payload 824 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 -- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc114138d40 msgr2=0x7fc11413f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc114138d40 0x7fc11413f500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 -- 172.21.15.6:0/2905393464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114132d00 msgr2=0x7fc114138330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114132d00 0x7fc114138330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient: _finish_hunting 0 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 monclient: found mon.noname-a 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:09.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc119def700 1 -- 172.21.15.6:0/2905393464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc10c009a20 con 0x7fc114098670 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc11a5f0700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc114138d40 0x7fc11413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.793+0000 7fc1195ee700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114132d00 0x7fc114138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc119def700 1 -- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 msgr2=0x7fc114141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc119def700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 0x7fc114141600 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c5e0 tx=0x7fc110004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc110005d40 con 0x7fc114098670 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: dump: 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:09.807 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:09.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 5 monclient: authenticate success, global_id 4614 2023-09-30T00:48:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 10 monclient: _renew_subs 2023-09-30T00:48:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc1100070d0 con 0x7fc114098670 2023-09-30T00:48:09.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc114133810 con 0x7fc114098670 2023-09-30T00:48:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc110007490 con 0x7fc114098670 2023-09-30T00:48:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc109ffb700 10 monclient: dump: 2023-09-30T00:48:09.810 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:09.811 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:09.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc10bfff700 4 set_mon_vals no callback set 2023-09-30T00:48:09.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 10 monclient: _renew_subs 2023-09-30T00:48:09.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:09.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.796+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc11412e0e0 con 0x7fc114098670 2023-09-30T00:48:09.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc11001b3b0 con 0x7fc114098670 2023-09-30T00:48:09.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc109ffb700 1 --2- 172.21.15.6:0/2905393464 >> v2:172.21.15.22:6812/142603 conn(0x7fc0f80357a0 0x7fc0f8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:09.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc110046ea0 con 0x7fc114098670 2023-09-30T00:48:09.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc11c053700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:09.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc11c053700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc11c053700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.797+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc114072b00 con 0x7fc114098670 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.798+0000 7fc1195ee700 1 --2- 172.21.15.6:0/2905393464 >> v2:172.21.15.22:6812/142603 conn(0x7fc0f80357a0 0x7fc0f8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.798+0000 7fc1195ee700 10 monclient: get_auth_request con 0x7fc0f80357a0 auth_method 0 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.800+0000 7fc1195ee700 1 --2- 172.21.15.6:0/2905393464 >> v2:172.21.15.22:6812/142603 conn(0x7fc0f80357a0 0x7fc0f8037c50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc104007fd0 tx=0x7fc104009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.802+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc110017020 con 0x7fc114098670 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.802+0000 7fc109ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:09.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:09.802+0000 7fc109ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.049+0000 7fc11c053700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.049+0000 7fc11c053700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.049+0000 7fc11c053700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:10.050 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.049+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc114133d30 con 0x7fc114098670 2023-09-30T00:48:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.050+0000 7fc109ffb700 1 -- 172.21.15.6:0/2905393464 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc11001a360 con 0x7fc114098670 2023-09-30T00:48:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.050+0000 7fc109ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:10.051 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.050+0000 7fc109ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:10.051 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:10.051 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 >> v2:172.21.15.22:6812/142603 conn(0x7fc0f80357a0 msgr2=0x7fc0f8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 --2- 172.21.15.6:0/2905393464 >> v2:172.21.15.22:6812/142603 conn(0x7fc0f80357a0 0x7fc0f8037c50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fc104007fd0 tx=0x7fc104009040 comp rx=0 tx=0).stop 2023-09-30T00:48:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 10 monclient: shutdown 2023-09-30T00:48:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:10.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 msgr2=0x7fc114141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 0x7fc114141600 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc11000c5e0 tx=0x7fc110004a80 comp rx=0 tx=0).stop 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 shutdown_connections 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 --2- 172.21.15.6:0/2905393464 >> v2:172.21.15.22:6812/142603 conn(0x7fc0f80357a0 0x7fc0f8037c50 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc114138d40 0x7fc11413f500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc114098670 0x7fc114141600 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 --2- 172.21.15.6:0/2905393464 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc114132d00 0x7fc114138330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 >> 172.21.15.6:0/2905393464 conn(0x7fc1140a5480 msgr2=0x7fc1140a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 shutdown_connections 2023-09-30T00:48:10.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:10.052+0000 7fc11c053700 1 -- 172.21.15.6:0/2905393464 wait complete. 2023-09-30T00:48:13.067 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.206+0000 7f553b556700 10 monclient: get_monmap_and_config 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.206+0000 7f553b556700 10 monclient: build_initial_monmap 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.206+0000 7f553b556700 10 monclient: monmap: 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:13.207359+0000 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:13.207359+0000 2023-09-30T00:48:13.207 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:13.208 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.206+0000 7f553b556700 1 Processor -- start 2023-09-30T00:48:13.211 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.206+0000 7f553b556700 1 -- start start 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.206+0000 7f553b556700 10 monclient: init 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5534098670 0x7f5534132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): picked mon.noname-c con 0x7f5534098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 0x7f553412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): picked mon.noname-b con 0x7f5534098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5534098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f55340a1000 con 0x7f5534098e90 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:13.212 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f55340a0a50 con 0x7f5534098670 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f55340a0730 con 0x7f5534098a40 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f553b556700 10 monclient(hunting): authenticate will time out at 1489.505508s 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 0x7f553412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5539af3700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5534098e90 0x7f553412f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:51100/0 (socket says 172.21.15.6:51100) 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5539af3700 1 -- 172.21.15.6:0/1695616185 learned_addr learned my addr 172.21.15.6:0/1695616185 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): get_auth_request con 0x7f5534098a40 auth_method 0 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:13.213 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f55392f2700 1 --2- 172.21.15.6:0/1695616185 >> v2:172.21.15.22:3300/0 conn(0x7f5534098670 0x7f5534132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f55392f2700 10 monclient(hunting): get_auth_request con 0x7f5534098670 auth_method 0 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f55392f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f55392f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f55392f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 10 monclient(hunting): handle_auth_done global_id 4628 payload 274 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.207+0000 7f5538af1700 1 -- 172.21.15.6:0/1695616185 >> v2:172.21.15.22:3300/0 conn(0x7f5534098670 msgr2=0x7f5534132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 1 --2- 172.21.15.6:0/1695616185 >> v2:172.21.15.22:3300/0 conn(0x7f5534098670 0x7f5534132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.214 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 1 -- 172.21.15.6:0/1695616185 >> v1:172.21.15.6:6789/0 conn(0x7f5534098e90 legacy=0x7f553412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 10 monclient: _finish_hunting 0 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 1 monclient: found mon.noname-b 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f5538af1700 1 -- 172.21.15.6:0/1695616185 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5534068df0 con 0x7f5534098a40 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.208+0000 7f55392f2700 1 --2- 172.21.15.6:0/1695616185 >> v2:172.21.15.22:3300/0 conn(0x7f5534098670 0x7f5534132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f5538af1700 1 -- 172.21.15.6:0/1695616185 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 msgr2=0x7f553412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f5538af1700 1 --2- 172.21.15.6:0/1695616185 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 0x7f553412ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f55240098e0 tx=0x7f55240091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3a788b8e9679a12 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 1 -- 172.21.15.6:0/1695616185 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5524009e80 con 0x7f5534098a40 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:13.215 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: dump: 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:13.216 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: _finish_auth 0 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 1 -- 172.21.15.6:0/1695616185 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552400ba30 con 0x7f5534098a40 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 5 monclient: authenticate success, global_id 4628 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 1 -- 172.21.15.6:0/1695616185 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552400bdb0 con 0x7f5534098a40 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 4 set_mon_vals no callback set 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f552bfff700 10 monclient: dump: 2023-09-30T00:48:13.217 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 10 monclient: shutdown 2023-09-30T00:48:13.218 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 1 -- 172.21.15.6:0/1695616185 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 msgr2=0x7f553412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.210+0000 7f553b556700 1 --2- 172.21.15.6:0/1695616185 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 0x7f553412ede0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f55240098e0 tx=0x7f55240091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 1 -- 172.21.15.6:0/1695616185 shutdown_connections 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 1 --2- 172.21.15.6:0/1695616185 >> v2:172.21.15.22:3300/0 conn(0x7f5534098670 0x7f5534132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 1 --2- 172.21.15.6:0/1695616185 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098a40 0x7f553412ede0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 1 -- 172.21.15.6:0/1695616185 >> 172.21.15.6:0/1695616185 conn(0x7f55340a5480 msgr2=0x7f55340a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 1 -- 172.21.15.6:0/1695616185 shutdown_connections 2023-09-30T00:48:13.219 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 1 -- 172.21.15.6:0/1695616185 wait complete. 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 10 monclient: build_initial_monmap 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.211+0000 7f553b556700 10 monclient: monmap: 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:13.224 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.212+0000 7f553b556700 1 Processor -- start 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.212+0000 7f553b556700 1 -- start start 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.212+0000 7f553b556700 10 monclient: init 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5534098670 0x7f5534131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): picked mon.noname-b con 0x7f5534098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5534098a40 0x7f55341417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): picked mon.noname-a con 0x7f5534098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:13.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 0x7f5534141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): picked mon.noname-c con 0x7f5534098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5534130d50 con 0x7f5534098a40 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f553412f8a0 con 0x7f5534098670 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5534130790 con 0x7f5534098e90 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f553b556700 10 monclient(hunting): authenticate will time out at 1489.511445s 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 0x7f5534141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5534098a40 0x7f55341417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:13.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 0x7f5534141d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:44246/0 (socket says 172.21.15.6:44246) 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 -- 172.21.15.6:0/65353823 learned_addr learned my addr 172.21.15.6:0/65353823 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): get_auth_request con 0x7f5534098e90 auth_method 0 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): get_auth_request con 0x7f5534098a40 auth_method 0 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5534098670 0x7f5534131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:13.227 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): get_auth_request con 0x7f5534098670 auth_method 0 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5538af1700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f55392f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:13.228 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient(hunting): handle_auth_done global_id 4631 payload 824 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 -- 172.21.15.6:0/65353823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5534098670 msgr2=0x7f5534131710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5534098670 0x7f5534131710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 -- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5534098a40 msgr2=0x7f55341417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5534098a40 0x7f55341417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient: _finish_hunting 0 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 monclient: found mon.noname-c 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.213+0000 7f5539af3700 1 -- 172.21.15.6:0/65353823 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5524009420 con 0x7f5534098e90 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f5539af3700 1 -- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 msgr2=0x7f5534141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f5539af3700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 0x7f5534141d10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f552c00cbc0 tx=0x7f552c0053a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:13.229 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c00a9f0 con 0x7f5534098e90 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: dump: 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:13.230 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: _finish_auth 0 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 5 monclient: authenticate success, global_id 4631 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f552c00ab70 con 0x7f5534098e90 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f552c0070f0 con 0x7f5534098e90 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 10 monclient: _renew_subs 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5534130a70 con 0x7f5534098e90 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:13.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f552b7fe700 4 set_mon_vals no callback set 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f55297fa700 10 monclient: dump: 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:13.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 10 monclient: _renew_subs 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.216+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5534142250 con 0x7f5534098e90 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f552c0074e0 con 0x7f5534098e90 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f553b556700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f553b556700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f553b556700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5534072b00 con 0x7f5534098e90 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f55297fa700 1 --2- 172.21.15.6:0/65353823 >> v2:172.21.15.22:6812/142603 conn(0x7f55180357a0 0x7f5518037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f552c047c80 con 0x7f5534098e90 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f55392f2700 1 --2- 172.21.15.6:0/65353823 >> v2:172.21.15.22:6812/142603 conn(0x7f55180357a0 0x7f5518037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.217+0000 7f55392f2700 10 monclient: get_auth_request con 0x7f55180357a0 auth_method 0 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.220+0000 7f55392f2700 1 --2- 172.21.15.6:0/65353823 >> v2:172.21.15.22:6812/142603 conn(0x7f55180357a0 0x7f5518037c50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5530006f00 tx=0x7f5530008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:13.233 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.222+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f552c007790 con 0x7f5534098e90 2023-09-30T00:48:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.222+0000 7f55297fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:13.234 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.222+0000 7f55297fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.479+0000 7f553b556700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.479+0000 7f553b556700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.479+0000 7f553b556700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:13.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.479+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5534063920 con 0x7f5534098e90 2023-09-30T00:48:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.480+0000 7f55297fa700 1 -- 172.21.15.6:0/65353823 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f552c010300 con 0x7f5534098e90 2023-09-30T00:48:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.480+0000 7f55297fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:13.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.480+0000 7f55297fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:13.481 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:13.481 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 >> v2:172.21.15.22:6812/142603 conn(0x7f55180357a0 msgr2=0x7f5518037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 --2- 172.21.15.6:0/65353823 >> v2:172.21.15.22:6812/142603 conn(0x7f55180357a0 0x7f5518037c50 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f5530006f00 tx=0x7f5530008040 comp rx=0 tx=0).stop 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 10 monclient: shutdown 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 msgr2=0x7f5534141d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 0x7f5534141d10 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f552c00cbc0 tx=0x7f552c0053a0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 shutdown_connections 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 --2- 172.21.15.6:0/65353823 >> v2:172.21.15.22:6812/142603 conn(0x7f55180357a0 0x7f5518037c50 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.483 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5534098e90 0x7f5534141d10 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5534098670 0x7f5534131710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 --2- 172.21.15.6:0/65353823 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5534098a40 0x7f55341417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 >> 172.21.15.6:0/65353823 conn(0x7f55340a5480 msgr2=0x7f5534133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 shutdown_connections 2023-09-30T00:48:13.484 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:13.482+0000 7f553b556700 1 -- 172.21.15.6:0/65353823 wait complete. 2023-09-30T00:48:16.496 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:16.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.633+0000 7fdae5007700 10 monclient: get_monmap_and_config 2023-09-30T00:48:16.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.633+0000 7fdae5007700 10 monclient: build_initial_monmap 2023-09-30T00:48:16.634 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.633+0000 7fdae5007700 10 monclient: monmap: 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:16.634797+0000 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:16.634797+0000 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:16.635 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 Processor -- start 2023-09-30T00:48:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 -- start start 2023-09-30T00:48:16.636 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient: init 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient: _add_conns ranks=[2,4,1,3,0] 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 0x7fdae0132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): picked mon.noname-b con 0x7fdae0098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fdae0098a40 0x7fdae012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): picked mon.noname-c con 0x7fdae0098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdae0098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdae00a1000 con 0x7fdae0098e90 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:16.640 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdae00a0a50 con 0x7fdae0098a40 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae00a0730 con 0x7fdae0098670 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdae5007700 10 monclient(hunting): authenticate will time out at 1492.932826s 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.634+0000 7fdaded9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 0x7fdae0132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 0x7fdae0132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:46052/0 (socket says 172.21.15.6:46052) 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 -- 172.21.15.6:0/457516505 learned_addr learned my addr 172.21.15.6:0/457516505 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): get_auth_request con 0x7fdae0098670 auth_method 0 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:16.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 1 --2- 172.21.15.6:0/457516505 >> v2:172.21.15.22:3300/0 conn(0x7fdae0098a40 0x7fdae012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): get_auth_request con 0x7fdae0098a40 auth_method 0 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdade59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient(hunting): handle_auth_done global_id 4634 payload 274 2023-09-30T00:48:16.642 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 -- 172.21.15.6:0/457516505 >> v2:172.21.15.22:3300/0 conn(0x7fdae0098a40 msgr2=0x7fdae012ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 --2- 172.21.15.6:0/457516505 >> v2:172.21.15.22:3300/0 conn(0x7fdae0098a40 0x7fdae012ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 -- 172.21.15.6:0/457516505 >> v1:172.21.15.22:6789/0 conn(0x7fdae0098e90 legacy=0x7fdae012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient: _finish_hunting 0 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 monclient: found mon.noname-b 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.635+0000 7fdaded9d700 1 -- 172.21.15.6:0/457516505 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae0068df0 con 0x7fdae0098670 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaded9d700 1 -- 172.21.15.6:0/457516505 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 msgr2=0x7fdae0132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaded9d700 1 --2- 172.21.15.6:0/457516505 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 0x7fdae0132d00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdad00099b0 tx=0x7fdad0009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b1caa89a18e57c4 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 1 -- 172.21.15.6:0/457516505 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad0009ea0 con 0x7fdae0098670 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:16.643 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: dump: 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: _finish_auth 0 2023-09-30T00:48:16.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdae5007700 5 monclient: authenticate success, global_id 4634 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdae5007700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 1 -- 172.21.15.6:0/457516505 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad000ba30 con 0x7fdae0098670 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdae5007700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdae5007700 4 set_mon_vals no callback set 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 1 -- 172.21.15.6:0/457516505 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad000bdf0 con 0x7fdae0098670 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.637+0000 7fdaddd9b700 10 monclient: dump: 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:16.645 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 10 monclient: shutdown 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 -- 172.21.15.6:0/457516505 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 msgr2=0x7fdae0132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.646 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 --2- 172.21.15.6:0/457516505 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 0x7fdae0132d00 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fdad00099b0 tx=0x7fdad0009250 comp rx=0 tx=0).stop 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 -- 172.21.15.6:0/457516505 shutdown_connections 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 --2- 172.21.15.6:0/457516505 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098670 0x7fdae0132d00 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 --2- 172.21.15.6:0/457516505 >> v2:172.21.15.22:3300/0 conn(0x7fdae0098a40 0x7fdae012ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 -- 172.21.15.6:0/457516505 >> 172.21.15.6:0/457516505 conn(0x7fdae00a5480 msgr2=0x7fdae00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 -- 172.21.15.6:0/457516505 shutdown_connections 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.638+0000 7fdae5007700 1 -- 172.21.15.6:0/457516505 wait complete. 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 10 monclient: build_initial_monmap 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 10 monclient: monmap: 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:16.647 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 1 Processor -- start 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 1 -- start start 2023-09-30T00:48:16.648 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 10 monclient: init 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.639+0000 7fdae5007700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdae0098670 0x7fdae0141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): picked mon.noname-b con 0x7fdae0098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 0x7fdae012c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): picked mon.noname-a con 0x7fdae0098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:16.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 0x7fdae012c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): picked mon.noname-c con 0x7fdae0098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae0131300 con 0x7fdae0098a40 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdae01426e0 con 0x7fdae0098670 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdae009f290 con 0x7fdae0098e90 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdae5007700 10 monclient(hunting): authenticate will time out at 1492.938079s 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 0x7fdae012c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 0x7fdae012c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:16.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 0x7fdae012c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43634/0 (socket says 172.21.15.6:43634) 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 0x7fdae012c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:46068/0 (socket says 172.21.15.6:46068) 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 -- 172.21.15.6:0/2989391564 learned_addr learned my addr 172.21.15.6:0/2989391564 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): get_auth_request con 0x7fdae0098a40 auth_method 0 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): get_auth_request con 0x7fdae0098e90 auth_method 0 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:16.654 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdaded9d700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdae0098670 0x7fdae0141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdaded9d700 10 monclient(hunting): get_auth_request con 0x7fdae0098670 auth_method 0 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdaded9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdaded9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdaded9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient(hunting): handle_auth_done global_id 4620 payload 824 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 -- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 msgr2=0x7fdae012c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 0x7fdae012c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 -- 172.21.15.6:0/2989391564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdae0098670 msgr2=0x7fdae0141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdae0098670 0x7fdae0141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 monclient: found mon.noname-a 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdade59c700 1 -- 172.21.15.6:0/2989391564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdad00094f0 con 0x7fdae0098a40 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.640+0000 7fdadf59e700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 0x7fdae012c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.642+0000 7fdade59c700 1 -- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 msgr2=0x7fdae012c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.642+0000 7fdade59c700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 0x7fdae012c1a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdad400ec80 tx=0x7fdad40091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.642+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad400ca60 con 0x7fdae0098a40 2023-09-30T00:48:16.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.642+0000 7fdac6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.642+0000 7fdac6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.642+0000 7fdac6ffd700 10 monclient: dump: 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:16.657 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 5 monclient: authenticate success, global_id 4620 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdad400cbe0 con 0x7fdae0098a40 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 10 monclient: _renew_subs 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdae011c2f0 con 0x7fdae0098a40 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdad4013470 con 0x7fdae0098a40 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:16.658 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdac6ffd700 10 monclient: dump: 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 10 monclient: _renew_subs 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:16.659 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdae0141d90 con 0x7fdae0098a40 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.643+0000 7fdadd59a700 4 set_mon_vals no callback set 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdae5007700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdae5007700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdae5007700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fdad4013820 con 0x7fdae0098a40 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdae0072b00 con 0x7fdae0098a40 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdac6ffd700 1 --2- 172.21.15.6:0/2989391564 >> v2:172.21.15.22:6812/142603 conn(0x7fdacc0357d0 0x7fdacc037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fdad4047d70 con 0x7fdae0098a40 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdaded9d700 1 --2- 172.21.15.6:0/2989391564 >> v2:172.21.15.22:6812/142603 conn(0x7fdacc0357d0 0x7fdacc037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.644+0000 7fdaded9d700 10 monclient: get_auth_request con 0x7fdacc0357d0 auth_method 0 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.647+0000 7fdaded9d700 1 --2- 172.21.15.6:0/2989391564 >> v2:172.21.15.22:6812/142603 conn(0x7fdacc0357d0 0x7fdacc037c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdad00099b0 tx=0x7fdad0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:16.660 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.649+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fdad40236b0 con 0x7fdae0098a40 2023-09-30T00:48:16.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.649+0000 7fdac6ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:16.661 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.649+0000 7fdac6ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdae5007700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdae5007700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdae5007700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:16.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdae0063920 con 0x7fdae0098a40 2023-09-30T00:48:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdac6ffd700 1 -- 172.21.15.6:0/2989391564 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fdad4021070 con 0x7fdae0098a40 2023-09-30T00:48:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdac6ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:16.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.893+0000 7fdac6ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:16.895 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:16.895 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 >> v2:172.21.15.22:6812/142603 conn(0x7fdacc0357d0 msgr2=0x7fdacc037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.896 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 --2- 172.21.15.6:0/2989391564 >> v2:172.21.15.22:6812/142603 conn(0x7fdacc0357d0 0x7fdacc037c80 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fdad00099b0 tx=0x7fdad0016000 comp rx=0 tx=0).stop 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 10 monclient: shutdown 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 msgr2=0x7fdae012c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 0x7fdae012c1a0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fdad400ec80 tx=0x7fdad40091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 shutdown_connections 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 --2- 172.21.15.6:0/2989391564 >> v2:172.21.15.22:6812/142603 conn(0x7fdacc0357d0 0x7fdacc037c80 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdae0098e90 0x7fdae012c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdae0098670 0x7fdae0141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 --2- 172.21.15.6:0/2989391564 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdae0098a40 0x7fdae012c1a0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.895+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 >> 172.21.15.6:0/2989391564 conn(0x7fdae00a5480 msgr2=0x7fdae0130710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.896+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 shutdown_connections 2023-09-30T00:48:16.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:16.896+0000 7fdae5007700 1 -- 172.21.15.6:0/2989391564 wait complete. 2023-09-30T00:48:19.909 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:20.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.046+0000 7f1f254dd700 10 monclient: get_monmap_and_config 2023-09-30T00:48:20.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.046+0000 7f1f254dd700 10 monclient: build_initial_monmap 2023-09-30T00:48:20.047 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.046+0000 7f1f254dd700 10 monclient: monmap: 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:20.048074+0000 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:20.048074+0000 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:20.048 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.047+0000 7f1f254dd700 1 Processor -- start 2023-09-30T00:48:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.047+0000 7f1f254dd700 1 -- start start 2023-09-30T00:48:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.047+0000 7f1f254dd700 10 monclient: init 2023-09-30T00:48:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient: _add_conns ranks=[2,4,1,0,3] 2023-09-30T00:48:20.052 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 0x7f1f200989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): picked mon.noname-b con 0x7f1f20098600 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1f20132410 0x7f1f20132820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): picked mon.noname-c con 0x7f1f20132410 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1f20068df0 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f200a0f60 con 0x7f1f20068df0 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1f2006a280 con 0x7f1f20132410 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f20132d60 con 0x7f1f20098600 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:20.053 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f254dd700 10 monclient(hunting): authenticate will time out at 1496.346137s 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 0x7f1f200989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 0x7f1f200989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:46078/0 (socket says 172.21.15.6:46078) 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 -- 172.21.15.6:0/935305016 learned_addr learned my addr 172.21.15.6:0/935305016 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): get_auth_request con 0x7f1f20098600 auth_method 0 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 1 --2- 172.21.15.6:0/935305016 >> v2:172.21.15.22:3300/0 conn(0x7f1f20132410 0x7f1f20132820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:20.054 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): get_auth_request con 0x7f1f20132410 auth_method 0 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient(hunting): handle_auth_done global_id 4640 payload 274 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 -- 172.21.15.6:0/935305016 >> v2:172.21.15.22:3300/0 conn(0x7f1f20132410 msgr2=0x7f1f20132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 --2- 172.21.15.6:0/935305016 >> v2:172.21.15.22:3300/0 conn(0x7f1f20132410 0x7f1f20132820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.055 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 -- 172.21.15.6:0/935305016 >> v1:172.21.15.22:6789/0 conn(0x7f1f20068df0 legacy=0x7f1f20069270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient: _finish_hunting 0 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 monclient: found mon.noname-b 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.048+0000 7f1f1effd700 1 -- 172.21.15.6:0/935305016 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f200a1cb0 con 0x7f1f20098600 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.050+0000 7f1f1effd700 1 -- 172.21.15.6:0/935305016 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 msgr2=0x7f1f200989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.050+0000 7f1f1effd700 1 --2- 172.21.15.6:0/935305016 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 0x7f1f200989d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1f100099b0 tx=0x7f1f10009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=986d1d62e5968f83 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 1 -- 172.21.15.6:0/935305016 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10009ea0 con 0x7f1f20098600 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:20.056 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: dump: 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:20.057 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 1 -- 172.21.15.6:0/935305016 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f1000ba30 con 0x7f1f20098600 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 1 -- 172.21.15.6:0/935305016 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f1000bdf0 con 0x7f1f20098600 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 5 monclient: authenticate success, global_id 4640 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 4 set_mon_vals no callback set 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f1dffb700 10 monclient: dump: 2023-09-30T00:48:20.058 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 10 monclient: shutdown 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:20.059 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 -- 172.21.15.6:0/935305016 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 msgr2=0x7f1f200989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 --2- 172.21.15.6:0/935305016 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 0x7f1f200989d0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f1f100099b0 tx=0x7f1f10009250 comp rx=0 tx=0).stop 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 -- 172.21.15.6:0/935305016 shutdown_connections 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 --2- 172.21.15.6:0/935305016 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20098600 0x7f1f200989d0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 --2- 172.21.15.6:0/935305016 >> v2:172.21.15.22:3300/0 conn(0x7f1f20132410 0x7f1f20132820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 -- 172.21.15.6:0/935305016 >> 172.21.15.6:0/935305016 conn(0x7f1f200a54c0 msgr2=0x7f1f200a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 -- 172.21.15.6:0/935305016 shutdown_connections 2023-09-30T00:48:20.060 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.051+0000 7f1f254dd700 1 -- 172.21.15.6:0/935305016 wait complete. 2023-09-30T00:48:20.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.052+0000 7f1f254dd700 10 monclient: build_initial_monmap 2023-09-30T00:48:20.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.052+0000 7f1f254dd700 10 monclient: monmap: 2023-09-30T00:48:20.065 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:20.065 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:20.065 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:20.065 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 Processor -- start 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 -- start start 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient: init 2023-09-30T00:48:20.066 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 0x7f1f2012f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): picked mon.noname-a con 0x7f1f20068df0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1f20098600 0x7f1f2011bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): picked mon.noname-b con 0x7f1f20098600 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 0x7f1f2011c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): picked mon.noname-c con 0x7f1f20132410 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200a3ee0 con 0x7f1f20068df0 2023-09-30T00:48:20.067 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f200a1460 con 0x7f1f20098600 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1f2011fb90 con 0x7f1f20132410 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.053+0000 7f1f254dd700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f254dd700 10 monclient(hunting): authenticate will time out at 1496.352006s 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 0x7f1f2012f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 0x7f1f2011c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 0x7f1f2012f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43640/0 (socket says 172.21.15.6:43640) 2023-09-30T00:48:20.068 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 0x7f1f2011c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:46094/0 (socket says 172.21.15.6:46094) 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 -- 172.21.15.6:0/2660595038 learned_addr learned my addr 172.21.15.6:0/2660595038 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): get_auth_request con 0x7f1f20068df0 auth_method 0 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): get_auth_request con 0x7f1f20132410 auth_method 0 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:20.069 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1f20098600 0x7f1f2011bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): get_auth_request con 0x7f1f20098600 auth_method 0 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1e7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:20.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient(hunting): handle_auth_done global_id 4623 payload 824 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 -- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 msgr2=0x7f1f2011c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 0x7f1f2011c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 -- 172.21.15.6:0/2660595038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1f20098600 msgr2=0x7f1f2011bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1f20098600 0x7f1f2011bfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient: _finish_hunting 0 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 monclient: found mon.noname-a 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1effd700 1 -- 172.21.15.6:0/2660595038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f100094f0 con 0x7f1f20068df0 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.054+0000 7f1f1f7fe700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 0x7f1f2011c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f1effd700 1 -- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 msgr2=0x7f1f2012f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:20.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f1effd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 0x7f1f2012f660 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1f20132ea0 tx=0x7f1f10016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10006b80 con 0x7f1f20068df0 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: dump: 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:20.072 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1f10020070 con 0x7f1f20068df0 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f254dd700 5 monclient: authenticate success, global_id 4623 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f254dd700 10 monclient: _renew_subs 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f254dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f2012eb20 con 0x7f1f20068df0 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f10011d50 con 0x7f1f20068df0 2023-09-30T00:48:20.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f06ffd700 10 monclient: dump: 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:20.074 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.057+0000 7f1f254dd700 10 monclient: _renew_subs 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.056+0000 7f1f1d7fa700 4 set_mon_vals no callback set 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.057+0000 7f1f254dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.057+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f2011daa0 con 0x7f1f20068df0 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.057+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1f10022030 con 0x7f1f20068df0 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f254dd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f254dd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f254dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f200a5c00 con 0x7f1f20068df0 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f06ffd700 1 --2- 172.21.15.6:0/2660595038 >> v2:172.21.15.22:6812/142603 conn(0x7f1f0c0357d0 0x7f1f0c037c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1f10049520 con 0x7f1f20068df0 2023-09-30T00:48:20.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f1e7fc700 1 --2- 172.21.15.6:0/2660595038 >> v2:172.21.15.22:6812/142603 conn(0x7f1f0c0357d0 0x7f1f0c037c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:20.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.058+0000 7f1f1e7fc700 10 monclient: get_auth_request con 0x7f1f0c0357d0 auth_method 0 2023-09-30T00:48:20.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.061+0000 7f1f1e7fc700 1 --2- 172.21.15.6:0/2660595038 >> v2:172.21.15.22:6812/142603 conn(0x7f1f0c0357d0 0x7f1f0c037c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1f2011df40 tx=0x7f1f140091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:20.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.063+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f1f1001b660 con 0x7f1f20068df0 2023-09-30T00:48:20.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.063+0000 7f1f06ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:20.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.063+0000 7f1f06ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:20.313 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f254dd700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f254dd700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f254dd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1f200a5c00 con 0x7f1f20068df0 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f06ffd700 1 -- 172.21.15.6:0/2660595038 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f1f1002c380 con 0x7f1f20068df0 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f06ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.313+0000 7f1f06ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:20.314 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 >> v2:172.21.15.22:6812/142603 conn(0x7f1f0c0357d0 msgr2=0x7f1f0c037c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 --2- 172.21.15.6:0/2660595038 >> v2:172.21.15.22:6812/142603 conn(0x7f1f0c0357d0 0x7f1f0c037c80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1f2011df40 tx=0x7f1f140091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 10 monclient: shutdown 2023-09-30T00:48:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 msgr2=0x7f1f2012f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:20.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 0x7f1f2012f660 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1f20132ea0 tx=0x7f1f10016000 comp rx=0 tx=0).stop 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 shutdown_connections 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 --2- 172.21.15.6:0/2660595038 >> v2:172.21.15.22:6812/142603 conn(0x7f1f0c0357d0 0x7f1f0c037c80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1f20132410 0x7f1f2011c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1f20068df0 0x7f1f2012f660 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 --2- 172.21.15.6:0/2660595038 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1f20098600 0x7f1f2011bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 >> 172.21.15.6:0/2660595038 conn(0x7f1f200a54c0 msgr2=0x7f1f2009fd30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 shutdown_connections 2023-09-30T00:48:20.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:20.315+0000 7f1f254dd700 1 -- 172.21.15.6:0/2660595038 wait complete. 2023-09-30T00:48:23.330 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.469+0000 7f38c5c40700 10 monclient: get_monmap_and_config 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.469+0000 7f38c5c40700 10 monclient: build_initial_monmap 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.469+0000 7f38c5c40700 10 monclient: monmap: 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:23.470518+0000 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:23.470518+0000 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:23.470 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:23.471 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:23.471 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:23.471 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:23.471 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:23.471 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:23.471 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:23.474 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 Processor -- start 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 -- start start 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient: init 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f38c0098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 0x7f38c0137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): picked mon.noname-b con 0x7f38c0132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f38c0137b60 0x7f38c0139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): picked mon.noname-c con 0x7f38c0137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:23.475 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38c00a1000 con 0x7f38c0098670 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38c0099150 con 0x7f38c0137b60 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00a0a50 con 0x7f38c0132d00 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38c5c40700 10 monclient(hunting): authenticate will time out at 1499.768726s 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 0x7f38c0137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 0x7f38c0137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:46106/0 (socket says 172.21.15.6:46106) 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 1 -- 172.21.15.6:0/193986175 learned_addr learned my addr 172.21.15.6:0/193986175 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 10 monclient(hunting): get_auth_request con 0x7f38c0132d00 auth_method 0 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.476 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38beffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38bffff700 1 --2- 172.21.15.6:0/193986175 >> v2:172.21.15.22:3300/0 conn(0x7f38c0137b60 0x7f38c0139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.470+0000 7f38bffff700 10 monclient(hunting): get_auth_request con 0x7f38c0137b60 auth_method 0 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.477 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 10 monclient(hunting): handle_auth_done global_id 4646 payload 274 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 1 -- 172.21.15.6:0/193986175 >> v2:172.21.15.22:3300/0 conn(0x7f38c0137b60 msgr2=0x7f38c0139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 1 --2- 172.21.15.6:0/193986175 >> v2:172.21.15.22:3300/0 conn(0x7f38c0137b60 0x7f38c0139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 1 -- 172.21.15.6:0/193986175 >> v1:172.21.15.22:6789/0 conn(0x7f38c0098670 legacy=0x7f38c0098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 10 monclient: _finish_hunting 0 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 1 monclient: found mon.noname-b 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.471+0000 7f38beffd700 1 -- 172.21.15.6:0/193986175 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c00a0730 con 0x7f38c0132d00 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38beffd700 1 -- 172.21.15.6:0/193986175 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 msgr2=0x7f38c0137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:23.478 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38beffd700 1 --2- 172.21.15.6:0/193986175 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 0x7f38c0137620 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f38a80099b0 tx=0x7f38a8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cb54e88f3991d4a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 1 -- 172.21.15.6:0/193986175 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a8009ea0 con 0x7f38c0132d00 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: dump: 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:23.479 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 1 -- 172.21.15.6:0/193986175 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38a800ba30 con 0x7f38c0132d00 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38c5c40700 5 monclient: authenticate success, global_id 4646 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38c5c40700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38c5c40700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 1 -- 172.21.15.6:0/193986175 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38a800bdf0 con 0x7f38c0132d00 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38c5c40700 4 set_mon_vals no callback set 2023-09-30T00:48:23.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.473+0000 7f38be7fc700 10 monclient: dump: 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:23.481 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 10 monclient: shutdown 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 -- 172.21.15.6:0/193986175 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 msgr2=0x7f38c0137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 --2- 172.21.15.6:0/193986175 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 0x7f38c0137620 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f38a80099b0 tx=0x7f38a8009250 comp rx=0 tx=0).stop 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 -- 172.21.15.6:0/193986175 shutdown_connections 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 --2- 172.21.15.6:0/193986175 >> v2:172.21.15.22:3300/0 conn(0x7f38c0137b60 0x7f38c0139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 --2- 172.21.15.6:0/193986175 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0132d00 0x7f38c0137620 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 -- 172.21.15.6:0/193986175 >> 172.21.15.6:0/193986175 conn(0x7f38c00a5480 msgr2=0x7f38c00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 -- 172.21.15.6:0/193986175 shutdown_connections 2023-09-30T00:48:23.482 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 1 -- 172.21.15.6:0/193986175 wait complete. 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 10 monclient: build_initial_monmap 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.474+0000 7f38c5c40700 10 monclient: monmap: 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:23.487 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 Processor -- start 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 -- start start 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient: init 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:23.488 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 0x7f38c009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): picked mon.noname-c con 0x7f38c0098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38c0132d00 0x7f38c009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): picked mon.noname-b con 0x7f38c0132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0137b60 0x7f38c009d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): picked mon.noname-a con 0x7f38c0137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c00a1500 con 0x7f38c0137b60 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0137880 con 0x7f38c0132d00 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38c0133410 con 0x7f38c0098670 2023-09-30T00:48:23.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.475+0000 7f38c5c40700 10 monclient(hunting): authenticate will time out at 1499.773996s 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 0x7f38c009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0137b60 0x7f38c009d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 0x7f38c009c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:46108/0 (socket says 172.21.15.6:46108) 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 -- 172.21.15.6:0/3664044941 learned_addr learned my addr 172.21.15.6:0/3664044941 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): get_auth_request con 0x7f38c0098670 auth_method 0 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): get_auth_request con 0x7f38c0137b60 auth_method 0 2023-09-30T00:48:23.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38beffd700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38c0132d00 0x7f38c009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient(hunting): handle_auth_done global_id 4649 payload 824 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 -- 172.21.15.6:0/3664044941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38c0132d00 msgr2=0x7f38c009cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38c0132d00 0x7f38c009cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 -- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0137b60 msgr2=0x7f38c009d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0137b60 0x7f38c009d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 monclient: found mon.noname-c 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bf7fe700 1 -- 172.21.15.6:0/3664044941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a80094f0 con 0x7f38c0098670 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38beffd700 10 monclient: get_auth_request con 0x7f38c0132d00 auth_method 0 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38beffd700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38c0132d00 0x7f38c009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.476+0000 7f38bffff700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0137b60 0x7f38c009d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:23.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38bf7fe700 1 -- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 msgr2=0x7f38c009c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38bf7fe700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 0x7f38c009c7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c950 tx=0x7f38b8002760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8012840 con 0x7f38c0098670 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: dump: 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:23.493 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38b80129a0 con 0x7f38c0098670 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38c5c40700 5 monclient: authenticate success, global_id 4649 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38b8012d60 con 0x7f38c0098670 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:23.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38b77fe700 10 monclient: dump: 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38c5c40700 10 monclient: _renew_subs 2023-09-30T00:48:23.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38bdffb700 4 set_mon_vals no callback set 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38c5c40700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.478+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38c01336b0 con 0x7f38c0098670 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 10 monclient: _renew_subs 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38c009dc40 con 0x7f38c0098670 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f38b800a630 con 0x7f38c0098670 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.479+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38c009f7c0 con 0x7f38c0098670 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.480+0000 7f38b77fe700 1 --2- 172.21.15.6:0/3664044941 >> v2:172.21.15.22:6812/142603 conn(0x7f38a00357a0 0x7f38a0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:23.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.480+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f38b8047e50 con 0x7f38c0098670 2023-09-30T00:48:23.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.480+0000 7f38beffd700 1 --2- 172.21.15.6:0/3664044941 >> v2:172.21.15.22:6812/142603 conn(0x7f38a00357a0 0x7f38a0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:23.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.480+0000 7f38beffd700 10 monclient: get_auth_request con 0x7f38a00357a0 auth_method 0 2023-09-30T00:48:23.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.482+0000 7f38beffd700 1 --2- 172.21.15.6:0/3664044941 >> v2:172.21.15.22:6812/142603 conn(0x7f38a00357a0 0x7f38a0037c50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38c009f9a0 tx=0x7f38a8016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:23.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.484+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f38b80228f0 con 0x7f38c0098670 2023-09-30T00:48:23.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.485+0000 7f38b77fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:23.497 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.485+0000 7f38b77fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:23.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38c5c40700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38c5c40700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38c5c40700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f38c0072b00 con 0x7f38c0098670 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38b77fe700 1 -- 172.21.15.6:0/3664044941 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f38b8016070 con 0x7f38c0098670 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38b77fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.733+0000 7f38b77fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:23.734 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 >> v2:172.21.15.22:6812/142603 conn(0x7f38a00357a0 msgr2=0x7f38a0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 --2- 172.21.15.6:0/3664044941 >> v2:172.21.15.22:6812/142603 conn(0x7f38a00357a0 0x7f38a0037c50 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38c009f9a0 tx=0x7f38a8016000 comp rx=0 tx=0).stop 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 10 monclient: shutdown 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 msgr2=0x7f38c009c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 0x7f38c009c7a0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f38b800c950 tx=0x7f38b8002760 comp rx=0 tx=0).stop 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 shutdown_connections 2023-09-30T00:48:23.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 --2- 172.21.15.6:0/3664044941 >> v2:172.21.15.22:6812/142603 conn(0x7f38a00357a0 0x7f38a0037c50 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f38c0137b60 0x7f38c009d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f38c0098670 0x7f38c009c7a0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 --2- 172.21.15.6:0/3664044941 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f38c0132d00 0x7f38c009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 >> 172.21.15.6:0/3664044941 conn(0x7f38c00a5480 msgr2=0x7f38c0068ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 shutdown_connections 2023-09-30T00:48:23.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:23.735+0000 7f38c5c40700 1 -- 172.21.15.6:0/3664044941 wait complete. 2023-09-30T00:48:26.750 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:26.888 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.888+0000 7f417c0fc700 10 monclient: get_monmap_and_config 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.888+0000 7f417c0fc700 10 monclient: build_initial_monmap 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.888+0000 7f417c0fc700 10 monclient: monmap: 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:26.889467+0000 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:26.889467+0000 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:26.889 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:26.890 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:26.890 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:26.890 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:26.890 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.888+0000 7f417c0fc700 1 Processor -- start 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 1 -- start start 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient: init 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:48:26.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4174098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4174132d00 addr v1:172.21.15.6:6789/0 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4174138d40 0x7f417413b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient(hunting): picked mon.noname-c con 0x7f4174138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41740a1000 con 0x7f4174132d00 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f41740a0730 con 0x7f4174098670 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4174099150 con 0x7f4174138d40 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417c0fc700 10 monclient(hunting): authenticate will time out at 1503.187664s 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f4179697700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4174132d00 0x7f4174137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:34670/0 (socket says 172.21.15.6:34670) 2023-09-30T00:48:26.893 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f4179697700 1 -- 172.21.15.6:0/303911650 learned_addr learned my addr 172.21.15.6:0/303911650 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417a699700 1 --2- 172.21.15.6:0/303911650 >> v2:172.21.15.22:3300/0 conn(0x7f4174138d40 0x7f417413b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417a699700 10 monclient(hunting): get_auth_request con 0x7f4174138d40 auth_method 0 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417a699700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417a699700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.889+0000 7f417a699700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 10 monclient(hunting): handle_auth_done global_id 4648 payload 274 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 1 -- 172.21.15.6:0/303911650 >> v1:172.21.15.22:6789/0 conn(0x7f4174098670 legacy=0x7f4174098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 1 -- 172.21.15.6:0/303911650 >> v1:172.21.15.6:6789/0 conn(0x7f4174132d00 legacy=0x7f4174137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:26.894 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 10 monclient: _finish_hunting 0 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 1 monclient: found mon.noname-c 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:48:26.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.890+0000 7f417a699700 1 -- 172.21.15.6:0/303911650 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4174068df0 con 0x7f4174138d40 2023-09-30T00:48:26.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.892+0000 7f417a699700 1 -- 172.21.15.6:0/303911650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174138d40 msgr2=0x7f417413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.892+0000 7f417a699700 1 --2- 172.21.15.6:0/303911650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174138d40 0x7f417413b120 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f416c009e60 tx=0x7f416c009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f8a2b0e71fa9859 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 1 -- 172.21.15.6:0/303911650 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c00e070 con 0x7f4174138d40 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: dump: 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:26.901 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: _finish_auth 0 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 1 -- 172.21.15.6:0/303911650 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f416c012430 con 0x7f4174138d40 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 5 monclient: authenticate success, global_id 4648 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:26.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 1 -- 172.21.15.6:0/303911650 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f416c0127d0 con 0x7f4174138d40 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 4 set_mon_vals no callback set 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f4178e96700 10 monclient: dump: 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:26.903 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 10 monclient: shutdown 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 -- 172.21.15.6:0/303911650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174138d40 msgr2=0x7f417413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 --2- 172.21.15.6:0/303911650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174138d40 0x7f417413b120 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f416c009e60 tx=0x7f416c009780 comp rx=0 tx=0).stop 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 -- 172.21.15.6:0/303911650 shutdown_connections 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 --2- 172.21.15.6:0/303911650 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174138d40 0x7f417413b120 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 -- 172.21.15.6:0/303911650 >> 172.21.15.6:0/303911650 conn(0x7f41740a5480 msgr2=0x7f41740a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 -- 172.21.15.6:0/303911650 shutdown_connections 2023-09-30T00:48:26.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.893+0000 7f417c0fc700 1 -- 172.21.15.6:0/303911650 wait complete. 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.894+0000 7f417c0fc700 10 monclient: build_initial_monmap 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.894+0000 7f417c0fc700 10 monclient: monmap: 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:26.905 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.894+0000 7f417c0fc700 1 Processor -- start 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.894+0000 7f417c0fc700 1 -- start start 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.894+0000 7f417c0fc700 10 monclient: init 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 0x7f4174138430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): picked mon.noname-a con 0x7f4174098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174132d00 0x7f41741416b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): picked mon.noname-b con 0x7f4174132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4174138d40 0x7f417409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): picked mon.noname-c con 0x7f4174138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:26.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4174133a10 con 0x7f4174098670 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41741303b0 con 0x7f4174132d00 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4174133370 con 0x7f4174138d40 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417c0fc700 10 monclient(hunting): authenticate will time out at 1503.193192s 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 0x7f4174138430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 0x7f4174138430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57548/0 (socket says 172.21.15.6:57548) 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 -- 172.21.15.6:0/3567015330 learned_addr learned my addr 172.21.15.6:0/3567015330 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): get_auth_request con 0x7f4174098670 auth_method 0 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4174138d40 0x7f417409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): get_auth_request con 0x7f4174138d40 auth_method 0 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174132d00 0x7f41741416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): get_auth_request con 0x7f4174132d00 auth_method 0 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:26.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:26.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:26.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179697700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:26.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient(hunting): handle_auth_done global_id 4629 payload 824 2023-09-30T00:48:26.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 -- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4174138d40 msgr2=0x7f417409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4174138d40 0x7f417409c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 -- 172.21.15.6:0/3567015330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174132d00 msgr2=0x7f41741416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174132d00 0x7f41741416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient: _finish_hunting 0 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 monclient: found mon.noname-a 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f4179e98700 1 -- 172.21.15.6:0/3567015330 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f416c009a20 con 0x7f4174098670 2023-09-30T00:48:26.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.895+0000 7f417a699700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4174138d40 0x7f417409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.897+0000 7f4179e98700 1 -- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 msgr2=0x7f4174138430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.897+0000 7f4179e98700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 0x7f4174138430 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f417000c5d0 tx=0x7f4170007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41700117a0 con 0x7f4174098670 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: dump: 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.912 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 5 monclient: authenticate success, global_id 4629 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 10 monclient: _renew_subs 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4170011900 con 0x7f4174098670 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:26.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4174133690 con 0x7f4174098670 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4170011cc0 con 0x7f4174098670 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f4169ffb700 10 monclient: dump: 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:26.914 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f416bfff700 4 set_mon_vals no callback set 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 10 monclient: _renew_subs 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.898+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f417409e100 con 0x7f4174098670 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4170012630 con 0x7f4174098670 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f417c0fc700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f417c0fc700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f417c0fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:26.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4174133d30 con 0x7f4174098670 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f4169ffb700 1 --2- 172.21.15.6:0/3567015330 >> v2:172.21.15.22:6812/142603 conn(0x7f41580357a0 0x7f4158037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4170047510 con 0x7f4174098670 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f4179697700 1 --2- 172.21.15.6:0/3567015330 >> v2:172.21.15.22:6812/142603 conn(0x7f41580357a0 0x7f4158037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:26.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.899+0000 7f4179697700 10 monclient: get_auth_request con 0x7f41580357a0 auth_method 0 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.902+0000 7f4179697700 1 --2- 172.21.15.6:0/3567015330 >> v2:172.21.15.22:6812/142603 conn(0x7f41580357a0 0x7f4158037c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f417409c420 tx=0x7f4164007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.904+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4174133d30 con 0x7f4174098670 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.904+0000 7f4169ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:26.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:26.904+0000 7f4169ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.153+0000 7f417c0fc700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:27.154 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.153+0000 7f417c0fc700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.153+0000 7f417c0fc700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.153+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4174072b00 con 0x7f4174098670 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.154+0000 7f4169ffb700 1 -- 172.21.15.6:0/3567015330 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4170011e60 con 0x7f4174098670 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.154+0000 7f4169ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.154+0000 7f4169ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:27.155 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 >> v2:172.21.15.22:6812/142603 conn(0x7f41580357a0 msgr2=0x7f4158037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 --2- 172.21.15.6:0/3567015330 >> v2:172.21.15.22:6812/142603 conn(0x7f41580357a0 0x7f4158037c50 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f417409c420 tx=0x7f4164007f90 comp rx=0 tx=0).stop 2023-09-30T00:48:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 10 monclient: shutdown 2023-09-30T00:48:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 msgr2=0x7f4174138430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:27.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 0x7f4174138430 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f417000c5d0 tx=0x7f4170007f60 comp rx=0 tx=0).stop 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 shutdown_connections 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 --2- 172.21.15.6:0/3567015330 >> v2:172.21.15.22:6812/142603 conn(0x7f41580357a0 0x7f4158037c50 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4174138d40 0x7f417409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4174098670 0x7f4174138430 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 --2- 172.21.15.6:0/3567015330 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4174132d00 0x7f41741416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 >> 172.21.15.6:0/3567015330 conn(0x7f41740a5480 msgr2=0x7f4174130900 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 shutdown_connections 2023-09-30T00:48:27.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:27.156+0000 7f417c0fc700 1 -- 172.21.15.6:0/3567015330 wait complete. 2023-09-30T00:48:30.171 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:30.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.310+0000 7f9ab31cb700 10 monclient: get_monmap_and_config 2023-09-30T00:48:30.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.310+0000 7f9ab31cb700 10 monclient: build_initial_monmap 2023-09-30T00:48:30.311 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.310+0000 7f9ab31cb700 10 monclient: monmap: 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:30.312035+0000 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:30.312035+0000 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:30.312 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:30.313 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:30.313 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.311+0000 7f9ab31cb700 1 Processor -- start 2023-09-30T00:48:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.311+0000 7f9ab31cb700 1 -- start start 2023-09-30T00:48:30.316 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.311+0000 7f9ab31cb700 10 monclient: init 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 0x7f9aac132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): picked mon.noname-a con 0x7f9aac098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9aac098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 0x7f9aac12f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): picked mon.noname-b con 0x7f9aac098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9aac0a1000 con 0x7f9aac098a40 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9aac0a0a50 con 0x7f9aac098670 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.317 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0a0730 con 0x7f9aac098e90 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab31cb700 10 monclient(hunting): authenticate will time out at 1506.610353s 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 0x7f9aac12f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 0x7f9aac132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 0x7f9aac12f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:50580/0 (socket says 172.21.15.6:50580) 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 0x7f9aac132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:57562/0 (socket says 172.21.15.6:57562) 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 -- 172.21.15.6:0/2841033498 learned_addr learned my addr 172.21.15.6:0/2841033498 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): get_auth_request con 0x7f9aac098e90 auth_method 0 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.318 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): get_auth_request con 0x7f9aac098670 auth_method 0 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient(hunting): handle_auth_done global_id 4655 payload 274 2023-09-30T00:48:30.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 -- 172.21.15.6:0/2841033498 >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 msgr2=0x7f9aac132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 --2- 172.21.15.6:0/2841033498 >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 0x7f9aac132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 -- 172.21.15.6:0/2841033498 >> v1:172.21.15.22:6789/0 conn(0x7f9aac098a40 legacy=0x7f9aac12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient: _finish_hunting 0 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 monclient: found mon.noname-b 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab1768700 1 -- 172.21.15.6:0/2841033498 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac068df0 con 0x7f9aac098e90 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.312+0000 7f9ab0f67700 1 --2- 172.21.15.6:0/2841033498 >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 0x7f9aac132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab1768700 1 -- 172.21.15.6:0/2841033498 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 msgr2=0x7f9aac12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab1768700 1 --2- 172.21.15.6:0/2841033498 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 0x7f9aac12f4f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009f20 tx=0x7f9aa4009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f410acbb1549937 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 1 -- 172.21.15.6:0/2841033498 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa400e070 con 0x7f9aac098e90 2023-09-30T00:48:30.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: dump: 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:30.321 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: _finish_auth 0 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 1 -- 172.21.15.6:0/2841033498 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9aa400bba0 con 0x7f9aac098e90 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 5 monclient: authenticate success, global_id 4655 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 1 -- 172.21.15.6:0/2841033498 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9aa40092e0 con 0x7f9aac098e90 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 4 set_mon_vals no callback set 2023-09-30T00:48:30.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9aab7fe700 10 monclient: dump: 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:30.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 10 monclient: shutdown 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2841033498 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 msgr2=0x7f9aac12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2841033498 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 0x7f9aac12f4f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9aa4009f20 tx=0x7f9aa4009840 comp rx=0 tx=0).stop 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2841033498 shutdown_connections 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2841033498 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098e90 0x7f9aac12f4f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2841033498 >> v2:172.21.15.6:3300/0 conn(0x7f9aac098670 0x7f9aac132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2841033498 >> 172.21.15.6:0/2841033498 conn(0x7f9aac0a5480 msgr2=0x7f9aac0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2841033498 shutdown_connections 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.315+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2841033498 wait complete. 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.316+0000 7f9ab31cb700 10 monclient: build_initial_monmap 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.316+0000 7f9ab31cb700 10 monclient: monmap: 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:30.324 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:30.325 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:30.329 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.316+0000 7f9ab31cb700 1 Processor -- start 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 -- start start 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient: init 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 0x7f9aac12c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): picked mon.noname-c con 0x7f9aac098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9aac098a40 0x7f9aac09cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): picked mon.noname-b con 0x7f9aac098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac098e90 0x7f9aac09d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.330 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): picked mon.noname-a con 0x7f9aac098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac11cfb0 con 0x7f9aac098e90 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac11c6a0 con 0x7f9aac098a40 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0a1500 con 0x7f9aac098670 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab31cb700 10 monclient(hunting): authenticate will time out at 1506.615571s 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 0x7f9aac12c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 0x7f9aac12c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:50592/0 (socket says 172.21.15.6:50592) 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 1 -- 172.21.15.6:0/2816454319 learned_addr learned my addr 172.21.15.6:0/2816454319 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:30.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): get_auth_request con 0x7f9aac098670 auth_method 0 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac098e90 0x7f9aac09d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): get_auth_request con 0x7f9aac098e90 auth_method 0 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab1768700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9aabfff700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9aac098a40 0x7f9aac09cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9aabfff700 10 monclient(hunting): get_auth_request con 0x7f9aac098a40 auth_method 0 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9aabfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9aabfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9aabfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.317+0000 7f9ab0f67700 10 monclient(hunting): handle_auth_done global_id 4658 payload 824 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 1 -- 172.21.15.6:0/2816454319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9aac098a40 msgr2=0x7f9aac09cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9aac098a40 0x7f9aac09cdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 1 -- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac098e90 msgr2=0x7f9aac09d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac098e90 0x7f9aac09d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 10 monclient: _finish_hunting 0 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 1 monclient: found mon.noname-c 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab0f67700 1 -- 172.21.15.6:0/2816454319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a9c009420 con 0x7f9aac098670 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.318+0000 7f9ab1768700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac098e90 0x7f9aac09d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab0f67700 1 -- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 msgr2=0x7f9aac12c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab0f67700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 0x7f9aac12c730 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00ec80 tx=0x7f9a9c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c00cb00 con 0x7f9aac098670 2023-09-30T00:48:30.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: dump: 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:30.335 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: _finish_auth 0 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9a9c00cc60 con 0x7f9aac098670 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 5 monclient: authenticate success, global_id 4658 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 10 monclient: _renew_subs 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aa4009ae0 con 0x7f9aac098670 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c013470 con 0x7f9aac098670 2023-09-30T00:48:30.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aa8ff9700 10 monclient: dump: 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:30.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9aaaffd700 4 set_mon_vals no callback set 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 10 monclient: _renew_subs 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.320+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac09ea50 con 0x7f9aac098670 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9a9c0137e0 con 0x7f9aac098670 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9ab31cb700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9ab31cb700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9ab31cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac0a2a30 con 0x7f9aac098670 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9aa8ff9700 1 --2- 172.21.15.6:0/2816454319 >> v2:172.21.15.22:6812/142603 conn(0x7f9a8c0357a0 0x7f9a8c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.321+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9a9c047d50 con 0x7f9aac098670 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.322+0000 7f9aabfff700 1 --2- 172.21.15.6:0/2816454319 >> v2:172.21.15.22:6812/142603 conn(0x7f9a8c0357a0 0x7f9a8c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:30.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.322+0000 7f9aabfff700 10 monclient: get_auth_request con 0x7f9a8c0357a0 auth_method 0 2023-09-30T00:48:30.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.324+0000 7f9aabfff700 1 --2- 172.21.15.6:0/2816454319 >> v2:172.21.15.22:6812/142603 conn(0x7f9a8c0357a0 0x7f9a8c037c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9aac09e660 tx=0x7f9a94009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:30.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.326+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9a9c0236b0 con 0x7f9aac098670 2023-09-30T00:48:30.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.326+0000 7f9aa8ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:30.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.326+0000 7f9aa8ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:30.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.575+0000 7f9ab31cb700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:30.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.576+0000 7f9ab31cb700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.576+0000 7f9ab31cb700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.576+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9aac072b00 con 0x7f9aac098670 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.576+0000 7f9aa8ff9700 1 -- 172.21.15.6:0/2816454319 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f9a9c0232f0 con 0x7f9aac098670 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.576+0000 7f9aa8ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.576+0000 7f9aa8ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:30.577 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 >> v2:172.21.15.22:6812/142603 conn(0x7f9a8c0357a0 msgr2=0x7f9a8c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2816454319 >> v2:172.21.15.22:6812/142603 conn(0x7f9a8c0357a0 0x7f9a8c037c50 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f9aac09e660 tx=0x7f9a94009040 comp rx=0 tx=0).stop 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 10 monclient: shutdown 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 msgr2=0x7f9aac12c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 0x7f9aac12c730 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c00ec80 tx=0x7f9a9c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 shutdown_connections 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2816454319 >> v2:172.21.15.22:6812/142603 conn(0x7f9a8c0357a0 0x7f9a8c037c50 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9aac098e90 0x7f9aac09d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9aac098670 0x7f9aac12c730 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 --2- 172.21.15.6:0/2816454319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9aac098a40 0x7f9aac09cdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 >> 172.21.15.6:0/2816454319 conn(0x7f9aac0a5480 msgr2=0x7f9aac130b20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 shutdown_connections 2023-09-30T00:48:30.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:30.578+0000 7f9ab31cb700 1 -- 172.21.15.6:0/2816454319 wait complete. 2023-09-30T00:48:33.593 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:33.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.731+0000 7f5e72608700 10 monclient: get_monmap_and_config 2023-09-30T00:48:33.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.731+0000 7f5e72608700 10 monclient: build_initial_monmap 2023-09-30T00:48:33.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.731+0000 7f5e72608700 10 monclient: monmap: 2023-09-30T00:48:33.732 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:33.732 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:33.732840+0000 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:33.732840+0000 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:33.733 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 Processor -- start 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 -- start start 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient: init 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient: _add_conns ranks=[4,1,2,0,3] 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5e6c098670 0x7f5e6c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:33.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): picked mon.noname-c con 0x7f5e6c098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5e6c098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 0x7f5e6c12f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): picked mon.noname-b con 0x7f5e6c098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c0a1000 con 0x7f5e6c098a40 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c0a0a50 con 0x7f5e6c098670 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0a0730 con 0x7f5e6c098e90 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.732+0000 7f5e72608700 10 monclient(hunting): authenticate will time out at 1510.030940s 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 0x7f5e6c12f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:33.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 0x7f5e6c12f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:50600/0 (socket says 172.21.15.6:50600) 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 -- 172.21.15.6:0/1422290905 learned_addr learned my addr 172.21.15.6:0/1422290905 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): get_auth_request con 0x7f5e6c098e90 auth_method 0 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e6bfff700 1 --2- 172.21.15.6:0/1422290905 >> v2:172.21.15.22:3300/0 conn(0x7f5e6c098670 0x7f5e6c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e6bfff700 10 monclient(hunting): get_auth_request con 0x7f5e6c098670 auth_method 0 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e6bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:33.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e6bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e6bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_done global_id 4661 payload 274 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 -- 172.21.15.6:0/1422290905 >> v2:172.21.15.22:3300/0 conn(0x7f5e6c098670 msgr2=0x7f5e6c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 --2- 172.21.15.6:0/1422290905 >> v2:172.21.15.22:3300/0 conn(0x7f5e6c098670 0x7f5e6c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 -- 172.21.15.6:0/1422290905 >> v1:172.21.15.22:6789/0 conn(0x7f5e6c098a40 legacy=0x7f5e6c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient: _finish_hunting 0 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 monclient: found mon.noname-b 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e70ba5700 1 -- 172.21.15.6:0/1422290905 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c068df0 con 0x7f5e6c098e90 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.733+0000 7f5e6bfff700 1 --2- 172.21.15.6:0/1422290905 >> v2:172.21.15.22:3300/0 conn(0x7f5e6c098670 0x7f5e6c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:48:33.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e70ba5700 1 -- 172.21.15.6:0/1422290905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 msgr2=0x7f5e6c12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e70ba5700 1 --2- 172.21.15.6:0/1422290905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 0x7f5e6c12f4f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009f20 tx=0x7f5e64009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=988788bbf2282258 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 1 -- 172.21.15.6:0/1422290905 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e6400e070 con 0x7f5e6c098e90 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: dump: 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:33.741 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: _finish_auth 0 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 1 -- 172.21.15.6:0/1422290905 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6400bc50 con 0x7f5e6c098e90 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e72608700 5 monclient: authenticate success, global_id 4661 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e72608700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e72608700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 1 -- 172.21.15.6:0/1422290905 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e640092e0 con 0x7f5e6c098e90 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e72608700 4 set_mon_vals no callback set 2023-09-30T00:48:33.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.735+0000 7f5e6affd700 10 monclient: dump: 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:33.743 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 10 monclient: shutdown 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 -- 172.21.15.6:0/1422290905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 msgr2=0x7f5e6c12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 --2- 172.21.15.6:0/1422290905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 0x7f5e6c12f4f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f5e64009f20 tx=0x7f5e64009840 comp rx=0 tx=0).stop 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 -- 172.21.15.6:0/1422290905 shutdown_connections 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 --2- 172.21.15.6:0/1422290905 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098e90 0x7f5e6c12f4f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 --2- 172.21.15.6:0/1422290905 >> v2:172.21.15.22:3300/0 conn(0x7f5e6c098670 0x7f5e6c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 -- 172.21.15.6:0/1422290905 >> 172.21.15.6:0/1422290905 conn(0x7f5e6c0a5480 msgr2=0x7f5e6c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 -- 172.21.15.6:0/1422290905 shutdown_connections 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.736+0000 7f5e72608700 1 -- 172.21.15.6:0/1422290905 wait complete. 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.737+0000 7f5e72608700 10 monclient: build_initial_monmap 2023-09-30T00:48:33.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.737+0000 7f5e72608700 10 monclient: monmap: 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:33.745 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.737+0000 7f5e72608700 1 Processor -- start 2023-09-30T00:48:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.737+0000 7f5e72608700 1 -- start start 2023-09-30T00:48:33.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.737+0000 7f5e72608700 10 monclient: init 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 0x7f5e6c12c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): picked mon.noname-c con 0x7f5e6c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5e6c098a40 0x7f5e6c12c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): picked mon.noname-b con 0x7f5e6c098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c098e90 0x7f5e6c09cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): picked mon.noname-a con 0x7f5e6c098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c0a3b40 con 0x7f5e6c098e90 2023-09-30T00:48:33.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c12d7c0 con 0x7f5e6c098a40 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c12ddd0 con 0x7f5e6c098670 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e72608700 10 monclient(hunting): authenticate will time out at 1510.036276s 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 0x7f5e6c12c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 0x7f5e6c12c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:50614/0 (socket says 172.21.15.6:50614) 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 -- 172.21.15.6:0/4281993198 learned_addr learned my addr 172.21.15.6:0/4281993198 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): get_auth_request con 0x7f5e6c098670 auth_method 0 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:33.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c098e90 0x7f5e6c09cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): get_auth_request con 0x7f5e6c098e90 auth_method 0 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e70ba5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient(hunting): handle_auth_done global_id 4664 payload 824 2023-09-30T00:48:33.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6b7fe700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5e6c098a40 0x7f5e6c12c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 -- 172.21.15.6:0/4281993198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5e6c098a40 msgr2=0x7f5e6c12c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5e6c098a40 0x7f5e6c12c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 -- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c098e90 msgr2=0x7f5e6c09cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c098e90 0x7f5e6c09cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 monclient: found mon.noname-c 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6bfff700 1 -- 172.21.15.6:0/4281993198 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e64009ae0 con 0x7f5e6c098670 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6b7fe700 10 monclient: get_auth_request con 0x7f5e6c098a40 auth_method 0 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.738+0000 7f5e6b7fe700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5e6c098a40 0x7f5e6c12c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:48:33.753 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e6bfff700 1 -- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 msgr2=0x7f5e6c12c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e6bfff700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 0x7f5e6c12c280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400e8d0 tx=0x7f5e54008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54013620 con 0x7f5e6c098670 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: dump: 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:33.754 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: _finish_auth 0 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e54013780 con 0x7f5e6c098670 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 5 monclient: authenticate success, global_id 4664 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 10 monclient: _renew_subs 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c12d4b0 con 0x7f5e6c098670 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:33.755 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e54013b20 con 0x7f5e6c098670 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e63fff700 10 monclient: dump: 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:33.756 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e6a7fc700 4 set_mon_vals no callback set 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 10 monclient: _renew_subs 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.741+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c12e740 con 0x7f5e6c098670 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e72608700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e72608700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e72608700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c12d900 con 0x7f5e6c098670 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5e5400c3a0 con 0x7f5e6c098670 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e63fff700 1 --2- 172.21.15.6:0/4281993198 >> v2:172.21.15.22:6812/142603 conn(0x7f5e4c0357a0 0x7f5e4c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5e54047740 con 0x7f5e6c098670 2023-09-30T00:48:33.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e6b7fe700 1 --2- 172.21.15.6:0/4281993198 >> v2:172.21.15.22:6812/142603 conn(0x7f5e4c0357a0 0x7f5e4c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:33.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.742+0000 7f5e6b7fe700 10 monclient: get_auth_request con 0x7f5e4c0357a0 auth_method 0 2023-09-30T00:48:33.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.745+0000 7f5e6b7fe700 1 --2- 172.21.15.6:0/4281993198 >> v2:172.21.15.22:6812/142603 conn(0x7f5e4c0357a0 0x7f5e4c037c50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c09ea90 tx=0x7f5e5c009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:33.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.747+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5e54022c50 con 0x7f5e6c098670 2023-09-30T00:48:33.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.747+0000 7f5e63fff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:33.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.747+0000 7f5e63fff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e72608700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e72608700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e72608700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5e6c072b00 con 0x7f5e6c098670 2023-09-30T00:48:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e63fff700 1 -- 172.21.15.6:0/4281993198 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f5e54020020 con 0x7f5e6c098670 2023-09-30T00:48:33.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e63fff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:33.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.994+0000 7f5e63fff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:33.996 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:33.996 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 >> v2:172.21.15.22:6812/142603 conn(0x7f5e4c0357a0 msgr2=0x7f5e4c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 --2- 172.21.15.6:0/4281993198 >> v2:172.21.15.22:6812/142603 conn(0x7f5e4c0357a0 0x7f5e4c037c50 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e6c09ea90 tx=0x7f5e5c009040 comp rx=0 tx=0).stop 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 10 monclient: shutdown 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 msgr2=0x7f5e6c12c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 0x7f5e6c12c280 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5e5400e8d0 tx=0x7f5e54008040 comp rx=0 tx=0).stop 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 shutdown_connections 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 --2- 172.21.15.6:0/4281993198 >> v2:172.21.15.22:6812/142603 conn(0x7f5e4c0357a0 0x7f5e4c037c50 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5e6c098e90 0x7f5e6c09cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5e6c098670 0x7f5e6c12c280 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 --2- 172.21.15.6:0/4281993198 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5e6c098a40 0x7f5e6c12c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 >> 172.21.15.6:0/4281993198 conn(0x7f5e6c0a5480 msgr2=0x7f5e6c069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 shutdown_connections 2023-09-30T00:48:33.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:33.997+0000 7f5e72608700 1 -- 172.21.15.6:0/4281993198 wait complete. 2023-09-30T00:48:37.012 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.150+0000 7ff430b8e700 10 monclient: get_monmap_and_config 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.150+0000 7ff430b8e700 10 monclient: build_initial_monmap 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.150+0000 7ff430b8e700 10 monclient: monmap: 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:37.151255+0000 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:37.151255+0000 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:37.151 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.150+0000 7ff430b8e700 1 Processor -- start 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.150+0000 7ff430b8e700 1 -- start start 2023-09-30T00:48:37.152 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.150+0000 7ff430b8e700 10 monclient: init 2023-09-30T00:48:37.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:37.155 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient: _add_conns ranks=[0,3,1,4,2] 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff42c098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff42c132d00 0x7ff42c137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient(hunting): picked mon.noname-a con 0x7ff42c132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff42c137b60 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c0a1000 con 0x7ff42c098670 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff42c0a0730 con 0x7ff42c137b60 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff42c099150 con 0x7ff42c132d00 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff430b8e700 10 monclient(hunting): authenticate will time out at 1513.449460s 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff42c132d00 0x7ff42c137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff42a59c700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff42c098670 0x7ff42c098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53866/0 (socket says 172.21.15.6:53866) 2023-09-30T00:48:37.156 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff42c132d00 0x7ff42c137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50982/0 (socket says 172.21.15.6:50982) 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff42a59c700 1 -- 172.21.15.6:0/3661781334 learned_addr learned my addr 172.21.15.6:0/3661781334 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): get_auth_request con 0x7ff42c132d00 auth_method 0 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient(hunting): handle_auth_done global_id 4641 payload 274 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 1 -- 172.21.15.6:0/3661781334 >> v1:172.21.15.22:6789/0 conn(0x7ff42c137b60 legacy=0x7ff42c139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 1 -- 172.21.15.6:0/3661781334 >> v1:172.21.15.6:6789/0 conn(0x7ff42c098670 legacy=0x7ff42c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:37.157 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 1 monclient: found mon.noname-a 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.151+0000 7ff429d9b700 1 -- 172.21.15.6:0/3661781334 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff42c068df0 con 0x7ff42c132d00 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff429d9b700 1 -- 172.21.15.6:0/3661781334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 msgr2=0x7ff42c137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff429d9b700 1 --2- 172.21.15.6:0/3661781334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c137620 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff420009e60 tx=0x7ff420009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80fabe3a17d4fa9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 1 -- 172.21.15.6:0/3661781334 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff42000e070 con 0x7ff42c132d00 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: dump: 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:37.158 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: _finish_auth 0 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 1 -- 172.21.15.6:0/3661781334 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420012430 con 0x7ff42c132d00 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 5 monclient: authenticate success, global_id 4641 2023-09-30T00:48:37.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 4 set_mon_vals no callback set 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 1 -- 172.21.15.6:0/3661781334 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200127d0 con 0x7ff42c132d00 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff42959a700 10 monclient: dump: 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.160 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 10 monclient: shutdown 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 1 -- 172.21.15.6:0/3661781334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 msgr2=0x7ff42c137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 1 --2- 172.21.15.6:0/3661781334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c137620 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff420009e60 tx=0x7ff420009780 comp rx=0 tx=0).stop 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 1 -- 172.21.15.6:0/3661781334 shutdown_connections 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 1 --2- 172.21.15.6:0/3661781334 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c137620 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.154+0000 7ff430b8e700 1 -- 172.21.15.6:0/3661781334 >> 172.21.15.6:0/3661781334 conn(0x7ff42c0a5480 msgr2=0x7ff42c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 1 -- 172.21.15.6:0/3661781334 shutdown_connections 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 1 -- 172.21.15.6:0/3661781334 wait complete. 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 10 monclient: build_initial_monmap 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 10 monclient: monmap: 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:37.162 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:37.163 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:37.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 1 Processor -- start 2023-09-30T00:48:37.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 1 -- start start 2023-09-30T00:48:37.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.155+0000 7ff430b8e700 10 monclient: init 2023-09-30T00:48:37.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:37.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff42c098670 0x7ff42c12f640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): picked mon.noname-c con 0x7ff42c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c09c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): picked mon.noname-a con 0x7ff42c132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff42c137b60 0x7ff42c09cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): picked mon.noname-b con 0x7ff42c137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c0a1500 con 0x7ff42c132d00 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c137910 con 0x7ff42c137b60 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff42c09e750 con 0x7ff42c098670 2023-09-30T00:48:37.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff430b8e700 10 monclient(hunting): authenticate will time out at 1513.454436s 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c09c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c09c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50986/0 (socket says 172.21.15.6:50986) 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 -- 172.21.15.6:0/2375877272 learned_addr learned my addr 172.21.15.6:0/2375877272 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): get_auth_request con 0x7ff42c132d00 auth_method 0 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff42c098670 0x7ff42c12f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:37.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff42c137b60 0x7ff42c09cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): get_auth_request con 0x7ff42c098670 auth_method 0 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): get_auth_request con 0x7ff42c137b60 auth_method 0 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:37.170 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff42ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient(hunting): handle_auth_done global_id 4644 payload 824 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 -- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff42c098670 msgr2=0x7ff42c12f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff42c098670 0x7ff42c12f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 -- 172.21.15.6:0/2375877272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff42c137b60 msgr2=0x7ff42c09cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff42c137b60 0x7ff42c09cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 monclient: found mon.noname-a 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:37.171 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.156+0000 7ff429d9b700 1 -- 172.21.15.6:0/2375877272 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff420009a20 con 0x7ff42c132d00 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff429d9b700 1 -- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 msgr2=0x7ff42c09c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff429d9b700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c09c7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b6c0 tx=0x7ff420018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4200070e0 con 0x7ff42c132d00 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: dump: 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:37.172 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 5 monclient: authenticate success, global_id 4644 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff420021070 con 0x7ff42c132d00 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff420026430 con 0x7ff42c132d00 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 10 monclient: _renew_subs 2023-09-30T00:48:37.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff42c09e2d0 con 0x7ff42c132d00 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff428d99700 4 set_mon_vals no callback set 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff41a7fc700 10 monclient: dump: 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:37.174 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 10 monclient: _renew_subs 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.159+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42c136f90 con 0x7ff42c132d00 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff420023030 con 0x7ff42c132d00 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff430b8e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff430b8e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff430b8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42c072b00 con 0x7ff42c132d00 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff41a7fc700 1 --2- 172.21.15.6:0/2375877272 >> v2:172.21.15.22:6812/142603 conn(0x7ff40c0357a0 0x7ff40c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:37.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.160+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff4200499f0 con 0x7ff42c132d00 2023-09-30T00:48:37.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.161+0000 7ff42a59c700 1 --2- 172.21.15.6:0/2375877272 >> v2:172.21.15.22:6812/142603 conn(0x7ff40c0357a0 0x7ff40c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:37.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.161+0000 7ff42a59c700 10 monclient: get_auth_request con 0x7ff40c0357a0 auth_method 0 2023-09-30T00:48:37.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.163+0000 7ff42a59c700 1 --2- 172.21.15.6:0/2375877272 >> v2:172.21.15.22:6812/142603 conn(0x7ff40c0357a0 0x7ff40c037c50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff41c007880 tx=0x7ff41c007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:37.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.165+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff42001c420 con 0x7ff42c132d00 2023-09-30T00:48:37.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.165+0000 7ff41a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:37.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.165+0000 7ff41a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.414+0000 7ff430b8e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:37.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.414+0000 7ff430b8e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.414+0000 7ff430b8e700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.414+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff42c072b00 con 0x7ff42c132d00 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.415+0000 7ff41a7fc700 1 -- 172.21.15.6:0/2375877272 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7ff42001a3c0 con 0x7ff42c132d00 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.415+0000 7ff41a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.415+0000 7ff41a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:37.416 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 >> v2:172.21.15.22:6812/142603 conn(0x7ff40c0357a0 msgr2=0x7ff40c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 --2- 172.21.15.6:0/2375877272 >> v2:172.21.15.22:6812/142603 conn(0x7ff40c0357a0 0x7ff40c037c50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff41c007880 tx=0x7ff41c007f90 comp rx=0 tx=0).stop 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 10 monclient: shutdown 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 msgr2=0x7ff42c09c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c09c7a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff42000b6c0 tx=0x7ff420018000 comp rx=0 tx=0).stop 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 shutdown_connections 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 --2- 172.21.15.6:0/2375877272 >> v2:172.21.15.22:6812/142603 conn(0x7ff40c0357a0 0x7ff40c037c50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff42c137b60 0x7ff42c09cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.418 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff42c098670 0x7ff42c12f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 --2- 172.21.15.6:0/2375877272 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff42c132d00 0x7ff42c09c7a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 >> 172.21.15.6:0/2375877272 conn(0x7ff42c0a5480 msgr2=0x7ff42c131a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 shutdown_connections 2023-09-30T00:48:37.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:37.417+0000 7ff430b8e700 1 -- 172.21.15.6:0/2375877272 wait complete. 2023-09-30T00:48:40.431 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.569+0000 7f548fb9d700 10 monclient: get_monmap_and_config 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.569+0000 7f548fb9d700 10 monclient: build_initial_monmap 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.569+0000 7f548fb9d700 10 monclient: monmap: 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:40.570598+0000 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:40.570598+0000 2023-09-30T00:48:40.570 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:40.571 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.569+0000 7f548fb9d700 1 Processor -- start 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 1 -- start start 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient: init 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient: _add_conns ranks=[2,3,0,4,1] 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 0x7f5488130580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): picked mon.noname-b con 0x7f54880a3e50 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:40.575 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f548806f260 0x7f548806f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): picked mon.noname-a con 0x7f548806f260 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f548809dcb0 addr v1:172.21.15.6:6789/0 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f548806d030 con 0x7f548809dcb0 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54880a4220 con 0x7f548806f260 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f548806d770 con 0x7f54880a3e50 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548fb9d700 10 monclient(hunting): authenticate will time out at 1516.868761s 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d138700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f548806f260 0x7f548806f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:40.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d939700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 0x7f5488130580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548e13a700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f548809dcb0 0x7f548809e110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53870/0 (socket says 172.21.15.6:53870) 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548e13a700 1 -- 172.21.15.6:0/1598633519 learned_addr learned my addr 172.21.15.6:0/1598633519 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d939700 10 monclient(hunting): get_auth_request con 0x7f54880a3e50 auth_method 0 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d939700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d939700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d939700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d138700 10 monclient(hunting): get_auth_request con 0x7f548806f260 auth_method 0 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d138700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d138700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.570+0000 7f548d138700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:40.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d138700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d138700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d138700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 10 monclient(hunting): handle_auth_done global_id 4670 payload 274 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 1 -- 172.21.15.6:0/1598633519 >> v2:172.21.15.6:3300/0 conn(0x7f548806f260 msgr2=0x7f548806f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 1 --2- 172.21.15.6:0/1598633519 >> v2:172.21.15.6:3300/0 conn(0x7f548806f260 0x7f548806f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 1 -- 172.21.15.6:0/1598633519 >> v1:172.21.15.6:6789/0 conn(0x7f548809dcb0 legacy=0x7f548809e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 10 monclient: _finish_hunting 0 2023-09-30T00:48:40.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 1 monclient: found mon.noname-b 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d939700 1 -- 172.21.15.6:0/1598633519 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548806da50 con 0x7f54880a3e50 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.571+0000 7f548d138700 1 --2- 172.21.15.6:0/1598633519 >> v2:172.21.15.6:3300/0 conn(0x7f548806f260 0x7f548806f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548d939700 1 -- 172.21.15.6:0/1598633519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 msgr2=0x7f5488130580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548d939700 1 --2- 172.21.15.6:0/1598633519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 0x7f5488130580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f54840098d0 tx=0x7f5484009120 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4276f3fd953e8a8d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 1 -- 172.21.15.6:0/1598633519 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5484005e20 con 0x7f54880a3e50 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: dump: 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:40.579 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: _finish_auth 0 2023-09-30T00:48:40.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 1 -- 172.21.15.6:0/1598633519 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5484006030 con 0x7f54880a3e50 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548fb9d700 5 monclient: authenticate success, global_id 4670 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548fb9d700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 1 -- 172.21.15.6:0/1598633519 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f54840063d0 con 0x7f54880a3e50 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548c937700 10 monclient: dump: 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:40.581 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548fb9d700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.573+0000 7f548fb9d700 4 set_mon_vals no callback set 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 10 monclient: shutdown 2023-09-30T00:48:40.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 -- 172.21.15.6:0/1598633519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 msgr2=0x7f5488130580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 --2- 172.21.15.6:0/1598633519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 0x7f5488130580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f54840098d0 tx=0x7f5484009120 comp rx=0 tx=0).stop 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 -- 172.21.15.6:0/1598633519 shutdown_connections 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 --2- 172.21.15.6:0/1598633519 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f54880a3e50 0x7f5488130580 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 --2- 172.21.15.6:0/1598633519 >> v2:172.21.15.6:3300/0 conn(0x7f548806f260 0x7f548806f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 -- 172.21.15.6:0/1598633519 >> 172.21.15.6:0/1598633519 conn(0x7f54880a3340 msgr2=0x7f54880a3740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 -- 172.21.15.6:0/1598633519 shutdown_connections 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 1 -- 172.21.15.6:0/1598633519 wait complete. 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 10 monclient: build_initial_monmap 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.574+0000 7f548fb9d700 10 monclient: monmap: 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:40.583 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:40.584 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:40.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 Processor -- start 2023-09-30T00:48:40.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 -- start start 2023-09-30T00:48:40.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient: init 2023-09-30T00:48:40.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:40.588 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f548806f260 0x7f5488141970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): picked mon.noname-b con 0x7f548806f260 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 0x7f548809ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): picked mon.noname-c con 0x7f548809dcb0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54880a3e50 0x7f548812c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): picked mon.noname-a con 0x7f54880a3e50 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f548806a1b0 con 0x7f54880a3e50 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54881427d0 con 0x7f548806f260 2023-09-30T00:48:40.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5488068fe0 con 0x7f548809dcb0 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548fb9d700 10 monclient(hunting): authenticate will time out at 1516.873861s 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548d138700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 0x7f548809ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548e13a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54880a3e50 0x7f548812c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.575+0000 7f548e13a700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54880a3e50 0x7f548812c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51004/0 (socket says 172.21.15.6:51004) 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 0x7f548809ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34710/0 (socket says 172.21.15.6:34710) 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 1 -- 172.21.15.6:0/4142407495 learned_addr learned my addr 172.21.15.6:0/4142407495 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): get_auth_request con 0x7f54880a3e50 auth_method 0 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:40.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): get_auth_request con 0x7f548809dcb0 auth_method 0 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f548806f260 0x7f5488141970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:40.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548e13a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): get_auth_request con 0x7f548806f260 auth_method 0 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d939700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient(hunting): handle_auth_done global_id 4673 payload 824 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 -- 172.21.15.6:0/4142407495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f548806f260 msgr2=0x7f5488141970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f548806f260 0x7f5488141970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 -- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54880a3e50 msgr2=0x7f548812c100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54880a3e50 0x7f548812c100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient: _finish_hunting 0 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 monclient: found mon.noname-c 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.576+0000 7f548d138700 1 -- 172.21.15.6:0/4142407495 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5478009420 con 0x7f548809dcb0 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f548d138700 1 -- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 msgr2=0x7f548809ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f548d138700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 0x7f548809ca80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f547800ec80 tx=0x7f54780091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5478009700 con 0x7f548809dcb0 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:40.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: dump: 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:40.594 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547800ca60 con 0x7f548809dcb0 2023-09-30T00:48:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f548fb9d700 5 monclient: authenticate success, global_id 4673 2023-09-30T00:48:40.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f548fb9d700 10 monclient: _renew_subs 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f548fb9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54840093c0 con 0x7f548809dcb0 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f547800ce20 con 0x7f548809dcb0 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547dffb700 10 monclient: dump: 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:40.596 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.579+0000 7f548fb9d700 10 monclient: _renew_subs 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.578+0000 7f547ffff700 4 set_mon_vals no callback set 2023-09-30T00:48:40.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.579+0000 7f548fb9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.579+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f548812dc40 con 0x7f548809dcb0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.579+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5478013680 con 0x7f548809dcb0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f548fb9d700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f548fb9d700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f548fb9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f547dffb700 1 --2- 172.21.15.6:0/4142407495 >> v2:172.21.15.22:6812/142603 conn(0x7f546c0357a0 0x7f546c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f548806a010 con 0x7f548809dcb0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5478047c60 con 0x7f548809dcb0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f548d939700 1 --2- 172.21.15.6:0/4142407495 >> v2:172.21.15.22:6812/142603 conn(0x7f546c0357a0 0x7f546c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.580+0000 7f548d939700 10 monclient: get_auth_request con 0x7f546c0357a0 auth_method 0 2023-09-30T00:48:40.598 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.583+0000 7f548d939700 1 --2- 172.21.15.6:0/4142407495 >> v2:172.21.15.22:6812/142603 conn(0x7f546c0357a0 0x7f546c037c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f54840030c0 tx=0x7f5484015000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:40.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.585+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f548806a010 con 0x7f548809dcb0 2023-09-30T00:48:40.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.585+0000 7f547dffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:40.599 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.585+0000 7f547dffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.835+0000 7f548fb9d700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.835+0000 7f548fb9d700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:40.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.835+0000 7f548fb9d700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.835+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f548806a010 con 0x7f548809dcb0 2023-09-30T00:48:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.836+0000 7f547dffb700 1 -- 172.21.15.6:0/4142407495 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f548806a010 con 0x7f548809dcb0 2023-09-30T00:48:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.836+0000 7f547dffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:40.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.836+0000 7f547dffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:40.837 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:40.837 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 >> v2:172.21.15.22:6812/142603 conn(0x7f546c0357a0 msgr2=0x7f546c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 --2- 172.21.15.6:0/4142407495 >> v2:172.21.15.22:6812/142603 conn(0x7f546c0357a0 0x7f546c037c50 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f54840030c0 tx=0x7f5484015000 comp rx=0 tx=0).stop 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 10 monclient: shutdown 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 msgr2=0x7f548809ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 0x7f548809ca80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f547800ec80 tx=0x7f54780091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 shutdown_connections 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 --2- 172.21.15.6:0/4142407495 >> v2:172.21.15.22:6812/142603 conn(0x7f546c0357a0 0x7f546c037c50 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f54880a3e50 0x7f548812c100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f548806f260 0x7f5488141970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 --2- 172.21.15.6:0/4142407495 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f548809dcb0 0x7f548809ca80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 >> 172.21.15.6:0/4142407495 conn(0x7f54880a3340 msgr2=0x7f5488131400 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 shutdown_connections 2023-09-30T00:48:40.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:40.838+0000 7f548fb9d700 1 -- 172.21.15.6:0/4142407495 wait complete. 2023-09-30T00:48:43.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:43.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.990+0000 7f4dd867f700 10 monclient: get_monmap_and_config 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.990+0000 7f4dd867f700 10 monclient: build_initial_monmap 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.990+0000 7f4dd867f700 10 monclient: monmap: 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:43.992004+0000 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:43.992004+0000 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:43.992 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.991+0000 7f4dd867f700 1 Processor -- start 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.991+0000 7f4dd867f700 1 -- start start 2023-09-30T00:48:43.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.991+0000 7f4dd867f700 10 monclient: init 2023-09-30T00:48:44.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:44.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient: _add_conns ranks=[3,2,0,4,1] 2023-09-30T00:48:44.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 0x7f4dd0132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): picked mon.noname-a con 0x7f4dd0098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 0x7f4dd012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): picked mon.noname-b con 0x7f4dd0098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4dd0098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4dd00a1000 con 0x7f4dd0098e90 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4dd00a0a50 con 0x7f4dd0098670 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00a0730 con 0x7f4dd0098a40 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:44.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd867f700 10 monclient(hunting): authenticate will time out at 1520.290210s 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 0x7f4dd012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 0x7f4dd0132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 0x7f4dd0132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51020/0 (socket says 172.21.15.6:51020) 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 1 -- 172.21.15.6:0/2361276290 learned_addr learned my addr 172.21.15.6:0/2361276290 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 0x7f4dd012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:34716/0 (socket says 172.21.15.6:34716) 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): get_auth_request con 0x7f4dd0098670 auth_method 0 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd6c1c700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4dd0098e90 0x7f4dd012f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53886/0 (socket says 172.21.15.6:53886) 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): get_auth_request con 0x7f4dd0098a40 auth_method 0 2023-09-30T00:48:44.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient(hunting): handle_auth_done global_id 4676 payload 274 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 -- 172.21.15.6:0/2361276290 >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 msgr2=0x7f4dd0132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 --2- 172.21.15.6:0/2361276290 >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 0x7f4dd0132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:44.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 -- 172.21.15.6:0/2361276290 >> v1:172.21.15.6:6789/0 conn(0x7f4dd0098e90 legacy=0x7f4dd012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient: _finish_hunting 0 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 monclient: found mon.noname-b 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd5c1a700 1 -- 172.21.15.6:0/2361276290 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dd0068df0 con 0x7f4dd0098a40 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.992+0000 7f4dd641b700 1 --2- 172.21.15.6:0/2361276290 >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 0x7f4dd0132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.994+0000 7f4dd5c1a700 1 -- 172.21.15.6:0/2361276290 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 msgr2=0x7f4dd012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.994+0000 7f4dd5c1a700 1 --2- 172.21.15.6:0/2361276290 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 0x7f4dd012ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00098e0 tx=0x7f4dc00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94db64e4cc0ebd6e server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 1 -- 172.21.15.6:0/2361276290 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0011840 con 0x7f4dd0098a40 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: dump: 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:44.009 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: _finish_auth 0 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:44.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 1 -- 172.21.15.6:0/2361276290 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dc00119a0 con 0x7f4dd0098a40 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 5 monclient: authenticate success, global_id 4676 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 1 -- 172.21.15.6:0/2361276290 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dc0011d40 con 0x7f4dd0098a40 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 4 set_mon_vals no callback set 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd5419700 10 monclient: dump: 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:44.011 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 10 monclient: shutdown 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 -- 172.21.15.6:0/2361276290 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 msgr2=0x7f4dd012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 --2- 172.21.15.6:0/2361276290 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 0x7f4dd012ede0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00098e0 tx=0x7f4dc00091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:44.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 -- 172.21.15.6:0/2361276290 shutdown_connections 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 --2- 172.21.15.6:0/2361276290 >> v2:172.21.15.6:3300/0 conn(0x7f4dd0098670 0x7f4dd0132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 --2- 172.21.15.6:0/2361276290 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098a40 0x7f4dd012ede0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 -- 172.21.15.6:0/2361276290 >> 172.21.15.6:0/2361276290 conn(0x7f4dd00a5480 msgr2=0x7f4dd00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 -- 172.21.15.6:0/2361276290 shutdown_connections 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.995+0000 7f4dd867f700 1 -- 172.21.15.6:0/2361276290 wait complete. 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.996+0000 7f4dd867f700 10 monclient: build_initial_monmap 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.996+0000 7f4dd867f700 10 monclient: monmap: 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:44.013 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.996+0000 7f4dd867f700 1 Processor -- start 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.996+0000 7f4dd867f700 1 -- start start 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.996+0000 7f4dd867f700 10 monclient: init 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:44.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): picked mon.noname-a con 0x7f4dd0098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4dd0098a40 0x7f4dd012c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): picked mon.noname-b con 0x7f4dd0098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098e90 0x7f4dd012c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): picked mon.noname-c con 0x7f4dd0098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd009f3e0 con 0x7f4dd0098670 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd01313c0 con 0x7f4dd0098a40 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd00a1500 con 0x7f4dd0098e90 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd867f700 10 monclient(hunting): authenticate will time out at 1520.295417s 2023-09-30T00:48:44.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098e90 0x7f4dd012c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:51026/0 (socket says 172.21.15.6:51026) 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 -- 172.21.15.6:0/732054626 learned_addr learned my addr 172.21.15.6:0/732054626 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): get_auth_request con 0x7f4dd0098670 auth_method 0 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): get_auth_request con 0x7f4dd0098e90 auth_method 0 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:44.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd5c1a700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4dd0098a40 0x7f4dd012c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd5c1a700 10 monclient(hunting): get_auth_request con 0x7f4dd0098a40 auth_method 0 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd5c1a700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd5c1a700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd5c1a700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient(hunting): handle_auth_done global_id 4656 payload 824 2023-09-30T00:48:44.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 -- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098e90 msgr2=0x7f4dd012c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098e90 0x7f4dd012c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 -- 172.21.15.6:0/732054626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4dd0098a40 msgr2=0x7f4dd012c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4dd0098a40 0x7f4dd012c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 monclient: found mon.noname-a 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd641b700 1 -- 172.21.15.6:0/732054626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dcc009ae0 con 0x7f4dd0098670 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:43.997+0000 7f4dd6c1c700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098e90 0x7f4dd012c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd641b700 1 -- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 msgr2=0x7f4dd01418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd641b700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00c1a0 tx=0x7f4dcc00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:44.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc009dc0 con 0x7f4dd0098670 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: dump: 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:44.019 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4dcc00f040 con 0x7f4dd0098670 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 5 monclient: authenticate success, global_id 4656 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 10 monclient: _renew_subs 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dc0009420 con 0x7f4dd0098670 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc014480 con 0x7f4dd0098670 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:44.020 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dc67fc700 10 monclient: dump: 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:44.021 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd4c18700 4 set_mon_vals no callback set 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 10 monclient: _renew_subs 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.000+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd011c630 con 0x7f4dd0098670 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd641b700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00c1a0 tx=0x7f4dcc00b040 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f4dcc0480a0 osd_map(17..17 src has 1..17) v4 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4dcc0147f0 con 0x7f4dd0098670 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd867f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd867f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:44.022 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd867f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd0072b00 con 0x7f4dd0098670 2023-09-30T00:48:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dc67fc700 1 --2- 172.21.15.6:0/732054626 >> v2:172.21.15.22:6812/142603 conn(0x7f4db40357a0 0x7f4db4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd5c1a700 1 --2- 172.21.15.6:0/732054626 >> v2:172.21.15.22:6812/142603 conn(0x7f4db40357a0 0x7f4db4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.001+0000 7f4dd5c1a700 10 monclient: get_auth_request con 0x7f4db40357a0 auth_method 0 2023-09-30T00:48:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.004+0000 7f4dd5c1a700 1 --2- 172.21.15.6:0/732054626 >> v2:172.21.15.22:6812/142603 conn(0x7f4db40357a0 0x7f4db4037c50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00030c0 tx=0x7f4dc0016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:44.023 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.006+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4dcc0480a0 con 0x7f4dd0098670 2023-09-30T00:48:44.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.829+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4dcc01bb40 con 0x7f4dd0098670 2023-09-30T00:48:44.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.829+0000 7f4dc67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:44.830 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:44.829+0000 7f4dc67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:45.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.075+0000 7f4dd867f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.075+0000 7f4dd867f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.075+0000 7f4dd867f700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.075+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4dd011ca10 con 0x7f4dd0098670 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.076+0000 7f4dc67fc700 1 -- 172.21.15.6:0/732054626 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f4dcc01b310 con 0x7f4dd0098670 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.076+0000 7f4dc67fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.076+0000 7f4dc67fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:45.077 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 >> v2:172.21.15.22:6812/142603 conn(0x7f4db40357a0 msgr2=0x7f4db4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 --2- 172.21.15.6:0/732054626 >> v2:172.21.15.22:6812/142603 conn(0x7f4db40357a0 0x7f4db4037c50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00030c0 tx=0x7f4dc0016000 comp rx=0 tx=0).stop 2023-09-30T00:48:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 10 monclient: shutdown 2023-09-30T00:48:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:45.079 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 msgr2=0x7f4dd01418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f4dcc00c1a0 tx=0x7f4dcc00b040 comp rx=0 tx=0).stop 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 shutdown_connections 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 --2- 172.21.15.6:0/732054626 >> v2:172.21.15.22:6812/142603 conn(0x7f4db40357a0 0x7f4db4037c50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4dd0098e90 0x7f4dd012c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4dd0098670 0x7f4dd01418c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 --2- 172.21.15.6:0/732054626 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4dd0098a40 0x7f4dd012c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 >> 172.21.15.6:0/732054626 conn(0x7f4dd00a5480 msgr2=0x7f4dd0130920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 shutdown_connections 2023-09-30T00:48:45.080 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:45.078+0000 7f4dd867f700 1 -- 172.21.15.6:0/732054626 wait complete. 2023-09-30T00:48:48.093 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.230+0000 7fc0da686700 10 monclient: get_monmap_and_config 2023-09-30T00:48:48.231 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.230+0000 7fc0da686700 10 monclient: build_initial_monmap 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.230+0000 7fc0da686700 10 monclient: monmap: 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:48.232179+0000 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:48.232179+0000 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:48.232 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:48.233 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:48.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.231+0000 7fc0da686700 1 Processor -- start 2023-09-30T00:48:48.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.231+0000 7fc0da686700 1 -- start start 2023-09-30T00:48:48.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.231+0000 7fc0da686700 10 monclient: init 2023-09-30T00:48:48.242 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient: _add_conns ranks=[2,4,3,1,0] 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d4132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): picked mon.noname-b con 0x7fc0d4098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fc0d4098a40 0x7fc0d412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): picked mon.noname-c con 0x7fc0d4098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0d4098e90 0x7fc0d412f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): picked mon.noname-a con 0x7fc0d4098e90 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d40a0a50 con 0x7fc0d4098e90 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d40a1000 con 0x7fc0d4098a40 2023-09-30T00:48:48.243 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d40a0730 con 0x7fc0d4098670 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0da686700 10 monclient(hunting): authenticate will time out at 1524.530327s 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d4132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0d4098e90 0x7fc0d412f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fc0d4098e90 0x7fc0d412f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43934/0 (socket says 172.21.15.6:43934) 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d4132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36136/0 (socket says 172.21.15.6:36136) 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 -- 172.21.15.6:0/2989560494 learned_addr learned my addr 172.21.15.6:0/2989560494 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): get_auth_request con 0x7fc0d4098e90 auth_method 0 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:48.244 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): get_auth_request con 0x7fc0d4098670 auth_method 0 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:48.245 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d37fe700 1 --2- 172.21.15.6:0/2989560494 >> v2:172.21.15.22:3300/0 conn(0x7fc0d4098a40 0x7fc0d412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d37fe700 10 monclient(hunting): get_auth_request con 0x7fc0d4098a40 auth_method 0 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_done global_id 4659 payload 274 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 -- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 msgr2=0x7fc0d4132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 --2- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d4132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 -- 172.21.15.6:0/2989560494 >> v2:172.21.15.22:3300/0 conn(0x7fc0d4098a40 msgr2=0x7fc0d412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 --2- 172.21.15.6:0/2989560494 >> v2:172.21.15.22:3300/0 conn(0x7fc0d4098a40 0x7fc0d412ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient: _finish_hunting 0 2023-09-30T00:48:48.246 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 monclient: found mon.noname-a 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d8c23700 1 -- 172.21.15.6:0/2989560494 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d4068df0 con 0x7fc0d4098e90 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d3fff700 1 --2- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.232+0000 7fc0d37fe700 1 --2- 172.21.15.6:0/2989560494 >> v2:172.21.15.22:3300/0 conn(0x7fc0d4098a40 0x7fc0d412ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d8c23700 1 -- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 msgr2=0x7fc0d412f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d8c23700 1 --2- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d412f320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc0d406a400 tx=0x7fc0cc009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edcdcc3f0d050d46 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 1 -- 172.21.15.6:0/2989560494 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc00e070 con 0x7fc0d4098e90 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:48.247 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: dump: 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:48.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 1 -- 172.21.15.6:0/2989560494 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0cc00bc30 con 0x7fc0d4098e90 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 5 monclient: authenticate success, global_id 4659 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 1 -- 172.21.15.6:0/2989560494 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc0092e0 con 0x7fc0d4098e90 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 4 set_mon_vals no callback set 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0d2ffd700 10 monclient: dump: 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:48.249 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 10 monclient: shutdown 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 -- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 msgr2=0x7fc0d412f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:48.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 --2- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d412f320 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc0d406a400 tx=0x7fc0cc009840 comp rx=0 tx=0).stop 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 -- 172.21.15.6:0/2989560494 shutdown_connections 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 --2- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d412f320 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 --2- 172.21.15.6:0/2989560494 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d4132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 --2- 172.21.15.6:0/2989560494 >> v2:172.21.15.22:3300/0 conn(0x7fc0d4098a40 0x7fc0d412ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 -- 172.21.15.6:0/2989560494 >> 172.21.15.6:0/2989560494 conn(0x7fc0d40a5480 msgr2=0x7fc0d40a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.235+0000 7fc0da686700 1 -- 172.21.15.6:0/2989560494 shutdown_connections 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.236+0000 7fc0da686700 1 -- 172.21.15.6:0/2989560494 wait complete. 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.236+0000 7fc0da686700 10 monclient: build_initial_monmap 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.236+0000 7fc0da686700 10 monclient: monmap: 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:48.251 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.236+0000 7fc0da686700 1 Processor -- start 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 -- start start 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient: init 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:48:48.252 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): picked mon.noname-c con 0x7fc0d4098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc0d4098a40 0x7fc0d412c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): picked mon.noname-b con 0x7fc0d4098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d4099290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): picked mon.noname-a con 0x7fc0d4098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d409d1b0 con 0x7fc0d4098e90 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d409cb60 con 0x7fc0d4098a40 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d409ce40 con 0x7fc0d4098670 2023-09-30T00:48:48.253 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0da686700 10 monclient(hunting): authenticate will time out at 1524.535650s 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d4099290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36138/0 (socket says 172.21.15.6:36138) 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): get_auth_request con 0x7fc0d4098e90 auth_method 0 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 1 -- 172.21.15.6:0/2198157247 learned_addr learned my addr 172.21.15.6:0/2198157247 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): get_auth_request con 0x7fc0d4098670 auth_method 0 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:48.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d8c23700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.237+0000 7fc0d37fe700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc0d4098a40 0x7fc0d412c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d37fe700 10 monclient(hunting): get_auth_request con 0x7fc0d4098a40 auth_method 0 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:48.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 10 monclient(hunting): handle_auth_done global_id 4685 payload 824 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 1 -- 172.21.15.6:0/2198157247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc0d4098a40 msgr2=0x7fc0d412c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc0d4098a40 0x7fc0d412c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 1 -- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 msgr2=0x7fc0d4099290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d4099290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 10 monclient: _finish_hunting 0 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 1 monclient: found mon.noname-c 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d3fff700 1 -- 172.21.15.6:0/2198157247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c4009420 con 0x7fc0d4098670 2023-09-30T00:48:48.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.238+0000 7fc0d8c23700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d4099290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0d3fff700 1 -- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 msgr2=0x7fc0d412be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0d3fff700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400ec80 tx=0x7fc0c40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c400caa0 con 0x7fc0d4098670 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: dump: 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.257 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: _finish_auth 0 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c400cc00 con 0x7fc0d4098670 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 5 monclient: authenticate success, global_id 4685 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 10 monclient: _renew_subs 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:48.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0cc009ae0 con 0x7fc0d4098670 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4013470 con 0x7fc0d4098670 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0cbfff700 10 monclient: dump: 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:48.259 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0d27fc700 4 set_mon_vals no callback set 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 10 monclient: _renew_subs 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.240+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d409c7a0 con 0x7fc0d4098670 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0d3fff700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400ec80 tx=0x7fc0c40091a0 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7fc0c40137e0 mgrmap(e 5) v1 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fc0c40137e0 con 0x7fc0d4098670 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0da686700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0da686700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:48.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0da686700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:48.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d4097de0 con 0x7fc0d4098670 2023-09-30T00:48:48.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0cbfff700 1 --2- 172.21.15.6:0/2198157247 >> v2:172.21.15.22:6812/142603 conn(0x7fc0b40357a0 0x7fc0b4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:48.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.241+0000 7fc0d37fe700 1 --2- 172.21.15.6:0/2198157247 >> v2:172.21.15.22:6812/142603 conn(0x7fc0b40357a0 0x7fc0b4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:48.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.242+0000 7fc0d37fe700 10 monclient: get_auth_request con 0x7fc0b40357a0 auth_method 0 2023-09-30T00:48:48.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.244+0000 7fc0d37fe700 1 --2- 172.21.15.6:0/2198157247 >> v2:172.21.15.22:6812/142603 conn(0x7fc0b40357a0 0x7fc0b4037c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc006f00 tx=0x7fc0bc008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:48.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:48.246+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fc0c4047d30 con 0x7fc0d4098670 2023-09-30T00:48:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.070+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fc0c40238e0 con 0x7fc0d4098670 2023-09-30T00:48:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.070+0000 7fc0cbfff700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:49.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.070+0000 7fc0cbfff700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0da686700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0da686700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0da686700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc0d4072b00 con 0x7fc0d4098670 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0cbfff700 1 -- 172.21.15.6:0/2198157247 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fc0c40232f0 con 0x7fc0d4098670 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0cbfff700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.322+0000 7fc0cbfff700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:49.323 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:49.324 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 >> v2:172.21.15.22:6812/142603 conn(0x7fc0b40357a0 msgr2=0x7fc0b4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 --2- 172.21.15.6:0/2198157247 >> v2:172.21.15.22:6812/142603 conn(0x7fc0b40357a0 0x7fc0b4037c50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc006f00 tx=0x7fc0bc008040 comp rx=0 tx=0).stop 2023-09-30T00:48:49.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 10 monclient: shutdown 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 msgr2=0x7fc0d412be90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc0c400ec80 tx=0x7fc0c40091a0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 shutdown_connections 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 --2- 172.21.15.6:0/2198157247 >> v2:172.21.15.22:6812/142603 conn(0x7fc0b40357a0 0x7fc0b4037c50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fc0d4098e90 0x7fc0d4099290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fc0d4098670 0x7fc0d412be90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 --2- 172.21.15.6:0/2198157247 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fc0d4098a40 0x7fc0d412c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.324+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 >> 172.21.15.6:0/2198157247 conn(0x7fc0d40a5480 msgr2=0x7fc0d4069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.325+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 shutdown_connections 2023-09-30T00:48:49.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:49.325+0000 7fc0da686700 1 -- 172.21.15.6:0/2198157247 wait complete. 2023-09-30T00:48:52.341 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:52.480 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.480+0000 7f8467db0700 10 monclient: get_monmap_and_config 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.480+0000 7f8467db0700 10 monclient: build_initial_monmap 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.480+0000 7f8467db0700 10 monclient: monmap: 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:52.481473+0000 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:52.481473+0000 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:52.481 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:52.482 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:52.482 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:52.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.480+0000 7f8467db0700 1 Processor -- start 2023-09-30T00:48:52.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 1 -- start start 2023-09-30T00:48:52.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient: init 2023-09-30T00:48:52.489 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient: _add_conns ranks=[1,3,4,2,0] 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f8460098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8460132d00 0x7f8460137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): picked mon.noname-a con 0x7f8460132d00 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f8460137b60 0x7f8460139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): picked mon.noname-c con 0x7f8460137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:52.490 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84600a1000 con 0x7f8460098670 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8460099150 con 0x7f8460132d00 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84600a0a50 con 0x7f8460137b60 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f8467db0700 10 monclient(hunting): authenticate will time out at 1528.779761s 2023-09-30T00:48:52.491 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8460132d00 0x7f8460137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8460132d00 0x7f8460137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43948/0 (socket says 172.21.15.6:43948) 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 1 -- 172.21.15.6:0/3166111895 learned_addr learned my addr 172.21.15.6:0/3166111895 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 10 monclient(hunting): get_auth_request con 0x7f8460132d00 auth_method 0 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.481+0000 7f846534b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:52.492 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:52.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:52.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:52.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846634d700 1 --2- 172.21.15.6:0/3166111895 >> v2:172.21.15.22:3300/0 conn(0x7f8460137b60 0x7f8460139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:52.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846634d700 10 monclient(hunting): get_auth_request con 0x7f8460137b60 auth_method 0 2023-09-30T00:48:52.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846634d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:52.493 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846634d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:52.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846634d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:52.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 10 monclient(hunting): handle_auth_done global_id 4665 payload 274 2023-09-30T00:48:52.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 1 -- 172.21.15.6:0/3166111895 >> v2:172.21.15.22:3300/0 conn(0x7f8460137b60 msgr2=0x7f8460139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:52.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 1 --2- 172.21.15.6:0/3166111895 >> v2:172.21.15.22:3300/0 conn(0x7f8460137b60 0x7f8460139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 1 -- 172.21.15.6:0/3166111895 >> v1:172.21.15.22:6789/0 conn(0x7f8460098670 legacy=0x7f8460098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:52.494 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:52.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 1 monclient: found mon.noname-a 2023-09-30T00:48:52.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:52.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:52.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.482+0000 7f846534b700 1 -- 172.21.15.6:0/3166111895 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84600a0730 con 0x7f8460132d00 2023-09-30T00:48:52.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f846534b700 1 -- 172.21.15.6:0/3166111895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460132d00 msgr2=0x7f8460137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:52.495 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f846534b700 1 --2- 172.21.15.6:0/3166111895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460132d00 0x7f8460137620 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84600a3330 tx=0x7f8450009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed9bbe8be18a7e55 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:52.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 1 -- 172.21.15.6:0/3166111895 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845000ba30 con 0x7f8460132d00 2023-09-30T00:48:52.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:52.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:52.496 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: dump: 2023-09-30T00:48:52.496 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:52.496 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:52.497 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:52.498 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:52.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: _finish_auth 0 2023-09-30T00:48:52.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:52.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 1 -- 172.21.15.6:0/3166111895 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845000bb90 con 0x7f8460132d00 2023-09-30T00:48:52.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8467db0700 5 monclient: authenticate success, global_id 4665 2023-09-30T00:48:52.498 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8467db0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:52.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:52.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 1 -- 172.21.15.6:0/3166111895 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8450011420 con 0x7f8460132d00 2023-09-30T00:48:52.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8467db0700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:52.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8467db0700 4 set_mon_vals no callback set 2023-09-30T00:48:52.499 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.484+0000 7f8464b4a700 10 monclient: dump: 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.500 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:52.501 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:52.501 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:52.501 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:52.501 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:52.501 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:52.501 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 10 monclient: shutdown 2023-09-30T00:48:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 -- 172.21.15.6:0/3166111895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460132d00 msgr2=0x7f8460137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 --2- 172.21.15.6:0/3166111895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460132d00 0x7f8460137620 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f84600a3330 tx=0x7f8450009250 comp rx=0 tx=0).stop 2023-09-30T00:48:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 -- 172.21.15.6:0/3166111895 shutdown_connections 2023-09-30T00:48:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 --2- 172.21.15.6:0/3166111895 >> v2:172.21.15.22:3300/0 conn(0x7f8460137b60 0x7f8460139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.502 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 --2- 172.21.15.6:0/3166111895 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460132d00 0x7f8460137620 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 -- 172.21.15.6:0/3166111895 >> 172.21.15.6:0/3166111895 conn(0x7f84600a5480 msgr2=0x7f84600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:52.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 -- 172.21.15.6:0/3166111895 shutdown_connections 2023-09-30T00:48:52.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 1 -- 172.21.15.6:0/3166111895 wait complete. 2023-09-30T00:48:52.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 10 monclient: build_initial_monmap 2023-09-30T00:48:52.503 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.485+0000 7f8467db0700 10 monclient: monmap: 2023-09-30T00:48:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:52.504 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:52.505 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:52.505 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:52.505 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:52.505 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:52.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 Processor -- start 2023-09-30T00:48:52.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 -- start start 2023-09-30T00:48:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient: init 2023-09-30T00:48:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:48:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 0x7f8460136de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): picked mon.noname-a con 0x7f8460098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:52.506 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8460132d00 0x7f846009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:52.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): picked mon.noname-c con 0x7f8460132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:52.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8460137b60 0x7f846009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:52.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): picked mon.noname-b con 0x7f8460137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:52.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:52.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84601377a0 con 0x7f8460098670 2023-09-30T00:48:52.507 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846009e3b0 con 0x7f8460137b60 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f846009f830 con 0x7f8460132d00 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8467db0700 10 monclient(hunting): authenticate will time out at 1528.784800s 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 0x7f8460136de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:52.508 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f846534b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8460132d00 0x7f846009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 0x7f8460136de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:43956/0 (socket says 172.21.15.6:43956) 2023-09-30T00:48:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 1 -- 172.21.15.6:0/558745714 learned_addr learned my addr 172.21.15.6:0/558745714 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 10 monclient(hunting): get_auth_request con 0x7f8460098670 auth_method 0 2023-09-30T00:48:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:52.509 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f8465b4c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:52.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.486+0000 7f846534b700 10 monclient(hunting): get_auth_request con 0x7f8460132d00 auth_method 0 2023-09-30T00:48:52.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:52.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:52.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:52.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846634d700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8460137b60 0x7f846009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:52.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846634d700 10 monclient(hunting): get_auth_request con 0x7f8460137b60 auth_method 0 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846634d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846634d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846634d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 10 monclient(hunting): handle_auth_done global_id 4668 payload 824 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 1 -- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8460132d00 msgr2=0x7f846009c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8460132d00 0x7f846009c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 1 -- 172.21.15.6:0/558745714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8460137b60 msgr2=0x7f846009cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8460137b60 0x7f846009cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 10 monclient: _finish_hunting 0 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 1 monclient: found mon.noname-a 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f8465b4c700 1 -- 172.21.15.6:0/558745714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84500094f0 con 0x7f8460098670 2023-09-30T00:48:52.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846634d700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8460137b60 0x7f846009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:48:52.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.487+0000 7f846534b700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8460132d00 0x7f846009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8465b4c700 1 -- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 msgr2=0x7f8460136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8465b4c700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 0x7f8460136de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f845c00c5d0 tx=0x7f845c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c0117a0 con 0x7f8460098670 2023-09-30T00:48:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:52.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: dump: 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:52.517 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: _finish_auth 0 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8467db0700 5 monclient: authenticate success, global_id 4668 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f845c011900 con 0x7f8460098670 2023-09-30T00:48:52.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f845c011cc0 con 0x7f8460098670 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8467db0700 10 monclient: _renew_subs 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8467db0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8460133ad0 con 0x7f8460098670 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8457fff700 4 set_mon_vals no callback set 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:52.519 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:48:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.489+0000 7f8455ffb700 10 monclient: dump: 2023-09-30T00:48:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:52.520 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 10 monclient: _renew_subs 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846009d9b0 con 0x7f8460098670 2023-09-30T00:48:52.521 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f845c012630 con 0x7f8460098670 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8455ffb700 1 --2- 172.21.15.6:0/558745714 >> v2:172.21.15.22:6812/142603 conn(0x7f84440357a0 0x7f8444037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.490+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8460072b00 con 0x7f8460098670 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.491+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f845c0476c0 con 0x7f8460098670 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.491+0000 7f846534b700 1 --2- 172.21.15.6:0/558745714 >> v2:172.21.15.22:6812/142603 conn(0x7f84440357a0 0x7f8444037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.491+0000 7f846534b700 10 monclient: get_auth_request con 0x7f84440357a0 auth_method 0 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.494+0000 7f846534b700 1 --2- 172.21.15.6:0/558745714 >> v2:172.21.15.22:6812/142603 conn(0x7f84440357a0 0x7f8444037c50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84600a3330 tx=0x7f8450016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.495+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f845c01c460 con 0x7f8460098670 2023-09-30T00:48:52.522 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.495+0000 7f8455ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:52.523 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.495+0000 7f8455ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:52.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8467db0700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8467db0700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8467db0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f846009f5e0 con 0x7f8460098670 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8455ffb700 1 -- 172.21.15.6:0/558745714 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f845c0102e0 con 0x7f8460098670 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8455ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.740+0000 7f8455ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:52.741 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:52.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 >> v2:172.21.15.22:6812/142603 conn(0x7f84440357a0 msgr2=0x7f8444037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:52.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 --2- 172.21.15.6:0/558745714 >> v2:172.21.15.22:6812/142603 conn(0x7f84440357a0 0x7f8444037c50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f84600a3330 tx=0x7f8450016000 comp rx=0 tx=0).stop 2023-09-30T00:48:52.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 10 monclient: shutdown 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 msgr2=0x7f8460136de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 0x7f8460136de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f845c00c5d0 tx=0x7f845c007f60 comp rx=0 tx=0).stop 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 shutdown_connections 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 --2- 172.21.15.6:0/558745714 >> v2:172.21.15.22:6812/142603 conn(0x7f84440357a0 0x7f8444037c50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8460137b60 0x7f846009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8460098670 0x7f8460136de0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 --2- 172.21.15.6:0/558745714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8460132d00 0x7f846009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.742+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 >> 172.21.15.6:0/558745714 conn(0x7f84600a5480 msgr2=0x7f8460131930 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.743+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 shutdown_connections 2023-09-30T00:48:52.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:52.743+0000 7f8467db0700 1 -- 172.21.15.6:0/558745714 wait complete. 2023-09-30T00:48:55.757 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.894+0000 7f5803665700 10 monclient: get_monmap_and_config 2023-09-30T00:48:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.894+0000 7f5803665700 10 monclient: build_initial_monmap 2023-09-30T00:48:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.894+0000 7f5803665700 10 monclient: monmap: 2023-09-30T00:48:55.895 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:55.896022+0000 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:55.896022+0000 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:55.896 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:55.897 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.895+0000 7f5803665700 1 Processor -- start 2023-09-30T00:48:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.895+0000 7f5803665700 1 -- start start 2023-09-30T00:48:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.895+0000 7f5803665700 10 monclient: init 2023-09-30T00:48:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient: _add_conns ranks=[4,2,3,1,0] 2023-09-30T00:48:55.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f57fc098570 0x7f57fc098940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): picked mon.noname-c con 0x7f57fc098570 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc12ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): picked mon.noname-b con 0x7f57fc098e80 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f57fc132ca0 0x7f57fc12f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): picked mon.noname-a con 0x7f57fc132ca0 addr v2:172.21.15.6:3300/0 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc0a08c0 con 0x7f57fc132ca0 2023-09-30T00:48:55.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57fc0a1bf0 con 0x7f57fc098570 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc096a80 con 0x7f57fc098e80 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5803665700 10 monclient(hunting): authenticate will time out at 1532.194841s 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.896+0000 7f5800c00700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc12ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc12ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36148/0 (socket says 172.21.15.6:36148) 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 -- 172.21.15.6:0/1843720067 learned_addr learned my addr 172.21.15.6:0/1843720067 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): get_auth_request con 0x7f57fc098e80 auth_method 0 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:55.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 1 --2- 172.21.15.6:0/1843720067 >> v2:172.21.15.6:3300/0 conn(0x7f57fc132ca0 0x7f57fc12f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): get_auth_request con 0x7f57fc132ca0 auth_method 0 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 1 --2- 172.21.15.6:0/1843720067 >> v2:172.21.15.22:3300/0 conn(0x7f57fc098570 0x7f57fc098940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): get_auth_request con 0x7f57fc098570 auth_method 0 2023-09-30T00:48:55.904 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801c02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5801401700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient(hunting): handle_auth_done global_id 4691 payload 274 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 -- 172.21.15.6:0/1843720067 >> v2:172.21.15.22:3300/0 conn(0x7f57fc098570 msgr2=0x7f57fc098940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:55.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 --2- 172.21.15.6:0/1843720067 >> v2:172.21.15.22:3300/0 conn(0x7f57fc098570 0x7f57fc098940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 -- 172.21.15.6:0/1843720067 >> v2:172.21.15.6:3300/0 conn(0x7f57fc132ca0 msgr2=0x7f57fc12f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 --2- 172.21.15.6:0/1843720067 >> v2:172.21.15.6:3300/0 conn(0x7f57fc132ca0 0x7f57fc12f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient: _finish_hunting 0 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 monclient: found mon.noname-b 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.897+0000 7f5800c00700 1 -- 172.21.15.6:0/1843720067 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57fc096d60 con 0x7f57fc098e80 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f5800c00700 1 -- 172.21.15.6:0/1843720067 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 msgr2=0x7f57fc12ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f5800c00700 1 --2- 172.21.15.6:0/1843720067 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc12ec60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0a5360 tx=0x7f57f8009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9e4e4e09d4941ad1 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 1 -- 172.21.15.6:0/1843720067 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800b850 con 0x7f57fc098e80 2023-09-30T00:48:55.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: dump: 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:55.907 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: _finish_auth 0 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 1 -- 172.21.15.6:0/1843720067 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f800b9b0 con 0x7f57fc098e80 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f5803665700 5 monclient: authenticate success, global_id 4691 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f5803665700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 1 -- 172.21.15.6:0/1843720067 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f800bd70 con 0x7f57fc098e80 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f5803665700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f5803665700 4 set_mon_vals no callback set 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:55.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.899+0000 7f57f3fff700 10 monclient: dump: 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:55.909 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 10 monclient: shutdown 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 -- 172.21.15.6:0/1843720067 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 msgr2=0x7f57fc12ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 --2- 172.21.15.6:0/1843720067 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc12ec60 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f57fc0a5360 tx=0x7f57f8009250 comp rx=0 tx=0).stop 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 -- 172.21.15.6:0/1843720067 shutdown_connections 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 --2- 172.21.15.6:0/1843720067 >> v2:172.21.15.6:3300/0 conn(0x7f57fc132ca0 0x7f57fc12f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 --2- 172.21.15.6:0/1843720067 >> v2:172.21.15.22:3300/0 conn(0x7f57fc098570 0x7f57fc098940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 --2- 172.21.15.6:0/1843720067 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc12ec60 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 -- 172.21.15.6:0/1843720067 >> 172.21.15.6:0/1843720067 conn(0x7f57fc0a1eb0 msgr2=0x7f57fc0a22b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 -- 172.21.15.6:0/1843720067 shutdown_connections 2023-09-30T00:48:55.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.900+0000 7f5803665700 1 -- 172.21.15.6:0/1843720067 wait complete. 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.901+0000 7f5803665700 10 monclient: build_initial_monmap 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.901+0000 7f5803665700 10 monclient: monmap: 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:55.915 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.901+0000 7f5803665700 1 Processor -- start 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.901+0000 7f5803665700 1 -- start start 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.901+0000 7f5803665700 10 monclient: init 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57fc098570 0x7f57fc141be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): picked mon.noname-b con 0x7f57fc098570 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:55.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc099250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): picked mon.noname-c con 0x7f57fc098e80 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57fc132ca0 0x7f57fc099790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): picked mon.noname-a con 0x7f57fc132ca0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0a1170 con 0x7f57fc132ca0 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc131240 con 0x7f57fc098570 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc09b9a0 con 0x7f57fc098e80 2023-09-30T00:48:55.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5803665700 10 monclient(hunting): authenticate will time out at 1532.200648s 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc099250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc099250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:36150/0 (socket says 172.21.15.6:36150) 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57fc132ca0 0x7f57fc099790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 1 -- 172.21.15.6:0/1614419389 learned_addr learned my addr 172.21.15.6:0/1614419389 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): get_auth_request con 0x7f57fc098e80 auth_method 0 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): get_auth_request con 0x7f57fc132ca0 auth_method 0 2023-09-30T00:48:55.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5800c00700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801c02700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801401700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57fc098570 0x7f57fc141be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801401700 10 monclient(hunting): get_auth_request con 0x7f57fc098570 auth_method 0 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801401700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:55.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801401700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.902+0000 7f5801401700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 10 monclient(hunting): handle_auth_done global_id 4694 payload 824 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 1 -- 172.21.15.6:0/1614419389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57fc098570 msgr2=0x7f57fc141be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57fc098570 0x7f57fc141be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 1 -- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57fc132ca0 msgr2=0x7f57fc099790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57fc132ca0 0x7f57fc099790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 10 monclient: _finish_hunting 0 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 1 monclient: found mon.noname-c 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:55.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5800c00700 1 -- 172.21.15.6:0/1614419389 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f80094f0 con 0x7f57fc098e80 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.903+0000 7f5801c02700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57fc132ca0 0x7f57fc099790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5800c00700 1 -- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 msgr2=0x7f57fc099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5800c00700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc099250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009f70 tx=0x7f57f8017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f8011d70 con 0x7f57fc098e80 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: dump: 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.921 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: _finish_auth 0 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f8024430 con 0x7f57fc098e80 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 5 monclient: authenticate success, global_id 4694 2023-09-30T00:48:55.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 10 monclient: _renew_subs 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc11be30 con 0x7f57fc098e80 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57f80247f0 con 0x7f57fc098e80 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f17fa700 10 monclient: dump: 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.923 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 10 monclient: _renew_subs 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f57f37fe700 4 set_mon_vals no callback set 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.905+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc11c1c0 con 0x7f57fc098e80 2023-09-30T00:48:55.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.906+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f57f801a420 con 0x7f57fc098e80 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.906+0000 7f5803665700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.906+0000 7f5803665700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.906+0000 7f5803665700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.906+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc072b00 con 0x7f57fc098e80 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.906+0000 7f57f17fa700 1 --2- 172.21.15.6:0/1614419389 >> v2:172.21.15.22:6812/142603 conn(0x7f57e80357a0 0x7f57e8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.907+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f57f8048bb0 con 0x7f57fc098e80 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.907+0000 7f5801401700 1 --2- 172.21.15.6:0/1614419389 >> v2:172.21.15.22:6812/142603 conn(0x7f57e80357a0 0x7f57e8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.907+0000 7f5801401700 10 monclient: get_auth_request con 0x7f57e80357a0 auth_method 0 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.910+0000 7f5801401700 1 --2- 172.21.15.6:0/1614419389 >> v2:172.21.15.22:6812/142603 conn(0x7f57e80357a0 0x7f57e8037c50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009cb0 tx=0x7f57ec008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.911+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f57f802b710 con 0x7f57fc098e80 2023-09-30T00:48:55.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.911+0000 7f57f17fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:55.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:55.911+0000 7f57f17fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:56.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.158+0000 7f5803665700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:56.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.158+0000 7f5803665700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:56.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.158+0000 7f5803665700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:56.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.158+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f57fc11bfc0 con 0x7f57fc098e80 2023-09-30T00:48:56.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.159+0000 7f57f17fa700 1 -- 172.21.15.6:0/1614419389 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f57f8015070 con 0x7f57fc098e80 2023-09-30T00:48:56.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.159+0000 7f57f17fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:56.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.159+0000 7f57f17fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:56.160 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:56.160 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:56.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 >> v2:172.21.15.22:6812/142603 conn(0x7f57e80357a0 msgr2=0x7f57e8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:56.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 --2- 172.21.15.6:0/1614419389 >> v2:172.21.15.22:6812/142603 conn(0x7f57e80357a0 0x7f57e8037c50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009cb0 tx=0x7f57ec008040 comp rx=0 tx=0).stop 2023-09-30T00:48:56.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 10 monclient: shutdown 2023-09-30T00:48:56.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:56.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 msgr2=0x7f57fc099250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:56.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc099250 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f57f8009f70 tx=0x7f57f8017000 comp rx=0 tx=0).stop 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 shutdown_connections 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 --2- 172.21.15.6:0/1614419389 >> v2:172.21.15.22:6812/142603 conn(0x7f57e80357a0 0x7f57e8037c50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f57fc132ca0 0x7f57fc099790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f57fc098570 0x7f57fc141be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 --2- 172.21.15.6:0/1614419389 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f57fc098e80 0x7f57fc099250 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 >> 172.21.15.6:0/1614419389 conn(0x7f57fc0a1eb0 msgr2=0x7f57fc1318b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 shutdown_connections 2023-09-30T00:48:56.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:56.161+0000 7f5803665700 1 -- 172.21.15.6:0/1614419389 wait complete. 2023-09-30T00:48:59.175 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.313+0000 7f1834b8f700 10 monclient: get_monmap_and_config 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.313+0000 7f1834b8f700 10 monclient: build_initial_monmap 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.313+0000 7f1834b8f700 10 monclient: monmap: 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:48:59.314646+0000 2023-09-30T00:48:59.314 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:48:59.314646+0000 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:48:59.315 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 Processor -- start 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 -- start start 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient: init 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient: _add_conns ranks=[1,2,4,0,3] 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f1830098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:48:59.319 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f1830137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): picked mon.noname-b con 0x7f1830132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f1830137b60 0x7f1830139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): picked mon.noname-c con 0x7f1830137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300a1000 con 0x7f1830098670 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1830099150 con 0x7f1830137b60 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18300a0a50 con 0x7f1830132d00 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f1834b8f700 10 monclient(hunting): authenticate will time out at 1535.612835s 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.314+0000 7f182dd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f1830137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f1830137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40650/0 (socket says 172.21.15.6:40650) 2023-09-30T00:48:59.320 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 -- 172.21.15.6:0/2621329095 learned_addr learned my addr 172.21.15.6:0/2621329095 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): get_auth_request con 0x7f1830132d00 auth_method 0 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182ed9d700 1 --2- 172.21.15.6:0/2621329095 >> v2:172.21.15.22:3300/0 conn(0x7f1830137b60 0x7f1830139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182ed9d700 10 monclient(hunting): get_auth_request con 0x7f1830137b60 auth_method 0 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:59.321 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient(hunting): handle_auth_done global_id 4697 payload 274 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 -- 172.21.15.6:0/2621329095 >> v2:172.21.15.22:3300/0 conn(0x7f1830137b60 msgr2=0x7f1830139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 --2- 172.21.15.6:0/2621329095 >> v2:172.21.15.22:3300/0 conn(0x7f1830137b60 0x7f1830139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 -- 172.21.15.6:0/2621329095 >> v1:172.21.15.22:6789/0 conn(0x7f1830098670 legacy=0x7f1830098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 monclient: found mon.noname-b 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.315+0000 7f182dd9b700 1 -- 172.21.15.6:0/2621329095 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300a0730 con 0x7f1830132d00 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182dd9b700 1 -- 172.21.15.6:0/2621329095 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 msgr2=0x7f1830137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182dd9b700 1 --2- 172.21.15.6:0/2621329095 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f1830137620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f18240099b0 tx=0x7f1824009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a9cdb50bd91e34d6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 1 -- 172.21.15.6:0/2621329095 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182400ba30 con 0x7f1830132d00 2023-09-30T00:48:59.322 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: dump: 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:59.323 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: _finish_auth 0 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 1 -- 172.21.15.6:0/2621329095 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f182400bb90 con 0x7f1830132d00 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f1834b8f700 5 monclient: authenticate success, global_id 4697 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f1834b8f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f1834b8f700 10 monclient: get_monmap_and_config success 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 1 -- 172.21.15.6:0/2621329095 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824011420 con 0x7f1830132d00 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f1834b8f700 4 set_mon_vals no callback set 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.317+0000 7f182d59a700 10 monclient: dump: 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:59.324 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 10 monclient: shutdown 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 -- 172.21.15.6:0/2621329095 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 msgr2=0x7f1830137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:59.325 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2621329095 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f1830137620 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f18240099b0 tx=0x7f1824009250 comp rx=0 tx=0).stop 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 -- 172.21.15.6:0/2621329095 shutdown_connections 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2621329095 >> v2:172.21.15.22:3300/0 conn(0x7f1830137b60 0x7f1830139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2621329095 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f1830137620 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 -- 172.21.15.6:0/2621329095 >> 172.21.15.6:0/2621329095 conn(0x7f18300a5480 msgr2=0x7f18300a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 -- 172.21.15.6:0/2621329095 shutdown_connections 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 1 -- 172.21.15.6:0/2621329095 wait complete. 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 10 monclient: build_initial_monmap 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.318+0000 7f1834b8f700 10 monclient: monmap: 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:48:59.326 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:48:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:48:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:48:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:48:59.327 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:48:59.327 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:59.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 Processor -- start 2023-09-30T00:48:59.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 -- start start 2023-09-30T00:48:59.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient: init 2023-09-30T00:48:59.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient: _reopen_session rank -1 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1830098670 0x7f183012f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): picked mon.noname-b con 0x7f1830098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f183009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): picked mon.noname-c con 0x7f1830132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1830137b60 0x7f183009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): picked mon.noname-a con 0x7f1830137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18301364e0 con 0x7f1830137b60 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1830133780 con 0x7f1830098670 2023-09-30T00:48:59.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1830137780 con 0x7f1830132d00 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): _renew_subs 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f1834b8f700 10 monclient(hunting): authenticate will time out at 1535.617856s 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f182dd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f183009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.319+0000 7f182ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1830137b60 0x7f183009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f183009c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40660/0 (socket says 172.21.15.6:40660) 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 -- 172.21.15.6:0/2447860563 learned_addr learned my addr 172.21.15.6:0/2447860563 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): get_auth_request con 0x7f1830132d00 auth_method 0 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:59.333 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): get_auth_request con 0x7f1830137b60 auth_method 0 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182e59c700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1830098670 0x7f183012f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182e59c700 10 monclient(hunting): get_auth_request con 0x7f1830098670 auth_method 0 2023-09-30T00:48:59.334 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient(hunting): handle_auth_done global_id 4700 payload 824 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 -- 172.21.15.6:0/2447860563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1830098670 msgr2=0x7f183012f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1830098670 0x7f183012f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 -- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1830137b60 msgr2=0x7f183009cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1830137b60 0x7f183009cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 monclient: found mon.noname-c 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:48:59.335 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182dd9b700 1 -- 172.21.15.6:0/2447860563 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18240094f0 con 0x7f1830132d00 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.320+0000 7f182ed9d700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1830137b60 0x7f183009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f182dd9b700 1 -- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 msgr2=0x7f183009c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f182dd9b700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f183009c7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1824003080 tx=0x7f1824016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1824011e50 con 0x7f1830132d00 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: dump: 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.336 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1824020070 con 0x7f1830132d00 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 5 monclient: authenticate success, global_id 4700 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 10 monclient: _renew_subs 2023-09-30T00:48:59.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f183009f4e0 con 0x7f1830132d00 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f182401b450 con 0x7f1830132d00 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f181e7fc700 10 monclient: dump: 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:48:59.338 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f182cd99700 4 set_mon_vals no callback set 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 10 monclient: _renew_subs 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.322+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f183009e740 con 0x7f1830132d00 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.323+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f1824022030 con 0x7f1830132d00 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.323+0000 7f181e7fc700 1 --2- 172.21.15.6:0/2447860563 >> v2:172.21.15.22:6812/142603 conn(0x7f18100357a0 0x7f1810037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.323+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f1824049770 con 0x7f1830132d00 2023-09-30T00:48:59.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.324+0000 7f182e59c700 1 --2- 172.21.15.6:0/2447860563 >> v2:172.21.15.22:6812/142603 conn(0x7f18100357a0 0x7f1810037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.324+0000 7f1834b8f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.324+0000 7f1834b8f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.324+0000 7f1834b8f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.324+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1830133400 con 0x7f1830132d00 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.324+0000 7f182e59c700 10 monclient: get_auth_request con 0x7f18100357a0 auth_method 0 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.326+0000 7f182e59c700 1 --2- 172.21.15.6:0/2447860563 >> v2:172.21.15.22:6812/142603 conn(0x7f18100357a0 0x7f1810037c50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1820007fd0 tx=0x7f1820009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.329+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f182401a9c0 con 0x7f1830132d00 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.329+0000 7f181e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:48:59.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.329+0000 7f181e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f1834b8f700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f1834b8f700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f1834b8f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1830133400 con 0x7f1830132d00 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f181e7fc700 1 -- 172.21.15.6:0/2447860563 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f18240193f0 con 0x7f1830132d00 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f181e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:48:59.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.578+0000 7f181e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:48:59.580 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:48:59.580 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:48:59.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 >> v2:172.21.15.22:6812/142603 conn(0x7f18100357a0 msgr2=0x7f1810037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:59.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2447860563 >> v2:172.21.15.22:6812/142603 conn(0x7f18100357a0 0x7f1810037c50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1820007fd0 tx=0x7f1820009040 comp rx=0 tx=0).stop 2023-09-30T00:48:59.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 10 monclient: shutdown 2023-09-30T00:48:59.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 msgr2=0x7f183009c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f183009c7a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1824003080 tx=0x7f1824016000 comp rx=0 tx=0).stop 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 shutdown_connections 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2447860563 >> v2:172.21.15.22:6812/142603 conn(0x7f18100357a0 0x7f1810037c50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f1830137b60 0x7f183009cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f1830098670 0x7f183012f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 --2- 172.21.15.6:0/2447860563 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f1830132d00 0x7f183009c7a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 >> 172.21.15.6:0/2447860563 conn(0x7f18300a5480 msgr2=0x7f1830068ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 shutdown_connections 2023-09-30T00:48:59.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:48:59.580+0000 7f1834b8f700 1 -- 172.21.15.6:0/2447860563 wait complete. 2023-09-30T00:49:02.595 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:02.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.733+0000 7f74f7487700 10 monclient: get_monmap_and_config 2023-09-30T00:49:02.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.733+0000 7f74f7487700 10 monclient: build_initial_monmap 2023-09-30T00:49:02.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.733+0000 7f74f7487700 10 monclient: monmap: 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:02.734866+0000 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:02.734866+0000 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:02.735 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:02.736 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:02.736 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:02.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.734+0000 7f74f7487700 1 Processor -- start 2023-09-30T00:49:02.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.734+0000 7f74f7487700 1 -- start start 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.734+0000 7f74f7487700 10 monclient: init 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient: _add_conns ranks=[1,4,3,2,0] 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f74f0098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f74f0132d00 0x7f74f0137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): picked mon.noname-c con 0x7f74f0132d00 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74f0137b60 0x7f74f0139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): picked mon.noname-a con 0x7f74f0137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74f00a1000 con 0x7f74f0098670 2023-09-30T00:49:02.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f0099150 con 0x7f74f0137b60 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74f00a0a50 con 0x7f74f0132d00 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f7487700 10 monclient(hunting): authenticate will time out at 1539.033165s 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74f0137b60 0x7f74f0139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f74f0137b60 0x7f74f0139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:60408/0 (socket says 172.21.15.6:60408) 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 -- 172.21.15.6:0/2599603520 learned_addr learned my addr 172.21.15.6:0/2599603520 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:02.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): get_auth_request con 0x7f74f0137b60 auth_method 0 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f4a22700 1 --2- 172.21.15.6:0/2599603520 >> v2:172.21.15.22:3300/0 conn(0x7f74f0132d00 0x7f74f0137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f4a22700 10 monclient(hunting): get_auth_request con 0x7f74f0132d00 auth_method 0 2023-09-30T00:49:02.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f4a22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f4a22700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f4a22700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient(hunting): handle_auth_done global_id 4680 payload 274 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 -- 172.21.15.6:0/2599603520 >> v2:172.21.15.22:3300/0 conn(0x7f74f0132d00 msgr2=0x7f74f0137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 --2- 172.21.15.6:0/2599603520 >> v2:172.21.15.22:3300/0 conn(0x7f74f0132d00 0x7f74f0137620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 -- 172.21.15.6:0/2599603520 >> v1:172.21.15.22:6789/0 conn(0x7f74f0098670 legacy=0x7f74f0098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient: _finish_hunting 0 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 monclient: found mon.noname-a 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:02.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.735+0000 7f74f5a24700 1 -- 172.21.15.6:0/2599603520 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f00a0730 con 0x7f74f0137b60 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.737+0000 7f74f5a24700 1 -- 172.21.15.6:0/2599603520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0137b60 msgr2=0x7f74f0139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.737+0000 7f74f5a24700 1 --2- 172.21.15.6:0/2599603520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0137b60 0x7f74f0139f40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f74f0063790 tx=0x7f74e8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b997a6cc404f189 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.737+0000 7f74e7fff700 1 -- 172.21.15.6:0/2599603520 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e800e070 con 0x7f74f0137b60 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.737+0000 7f74e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: dump: 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:02.745 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: _finish_auth 0 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:02.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 5 monclient: authenticate success, global_id 4680 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 1 -- 172.21.15.6:0/2599603520 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74e800bc20 con 0x7f74f0137b60 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 4 set_mon_vals no callback set 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 1 -- 172.21.15.6:0/2599603520 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74e80092e0 con 0x7f74f0137b60 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74e7fff700 10 monclient: dump: 2023-09-30T00:49:02.747 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 10 monclient: shutdown 2023-09-30T00:49:02.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 -- 172.21.15.6:0/2599603520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0137b60 msgr2=0x7f74f0139f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 --2- 172.21.15.6:0/2599603520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0137b60 0x7f74f0139f40 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f74f0063790 tx=0x7f74e8009840 comp rx=0 tx=0).stop 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 -- 172.21.15.6:0/2599603520 shutdown_connections 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 --2- 172.21.15.6:0/2599603520 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0137b60 0x7f74f0139f40 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 --2- 172.21.15.6:0/2599603520 >> v2:172.21.15.22:3300/0 conn(0x7f74f0132d00 0x7f74f0137620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 -- 172.21.15.6:0/2599603520 >> 172.21.15.6:0/2599603520 conn(0x7f74f00a5480 msgr2=0x7f74f00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 -- 172.21.15.6:0/2599603520 shutdown_connections 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.738+0000 7f74f7487700 1 -- 172.21.15.6:0/2599603520 wait complete. 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.739+0000 7f74f7487700 10 monclient: build_initial_monmap 2023-09-30T00:49:02.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.739+0000 7f74f7487700 10 monclient: monmap: 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:02.750 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:02.751 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 Processor -- start 2023-09-30T00:49:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 -- start start 2023-09-30T00:49:02.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient: init 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 0x7f74f012f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient(hunting): picked mon.noname-c con 0x7f74f0098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0132d00 0x7f74f009ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient(hunting): picked mon.noname-a con 0x7f74f0132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74f0137b60 0x7f74f01369f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:02.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient(hunting): picked mon.noname-b con 0x7f74f0137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f00a1500 con 0x7f74f0132d00 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.740+0000 7f74f7487700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74f0137570 con 0x7f74f0137b60 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f7487700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f7487700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74f009dc80 con 0x7f74f0098670 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f7487700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f7487700 10 monclient(hunting): authenticate will time out at 1539.039017s 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 0x7f74f012f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:02.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0132d00 0x7f74f009ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 0x7f74f012f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40674/0 (socket says 172.21.15.6:40674) 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 -- 172.21.15.6:0/3706217989 learned_addr learned my addr 172.21.15.6:0/3706217989 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): get_auth_request con 0x7f74f0098670 auth_method 0 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): get_auth_request con 0x7f74f0132d00 auth_method 0 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:02.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5a24700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74f0137b60 0x7f74f01369f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5a24700 10 monclient(hunting): get_auth_request con 0x7f74f0137b60 auth_method 0 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5a24700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5a24700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:02.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5a24700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient(hunting): handle_auth_done global_id 4703 payload 824 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 -- 172.21.15.6:0/3706217989 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74f0137b60 msgr2=0x7f74f01369f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74f0137b60 0x7f74f01369f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 -- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0132d00 msgr2=0x7f74f009ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0132d00 0x7f74f009ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient: _finish_hunting 0 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 monclient: found mon.noname-c 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:02.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f5223700 1 -- 172.21.15.6:0/3706217989 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e8009ae0 con 0x7f74f0098670 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.741+0000 7f74f4a22700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0132d00 0x7f74f009ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74f5223700 1 -- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 msgr2=0x7f74f012f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74f5223700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 0x7f74f012f110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00c600 tx=0x7f74ec007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec0117a0 con 0x7f74f0098670 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74e57fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74e57fa700 10 monclient: dump: 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:02.761 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:02.762 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74e57fa700 10 monclient: _finish_auth 0 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74e57fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74f7487700 5 monclient: authenticate success, global_id 4703 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74f7487700 10 monclient: _renew_subs 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.743+0000 7f74f7487700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74f009f230 con 0x7f74f0098670 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74ec011900 con 0x7f74f0098670 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f74ec011cc0 con 0x7f74f0098670 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:02.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 10 monclient: dump: 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:02.764 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e77fe700 4 set_mon_vals no callback set 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74f7487700 10 monclient: _renew_subs 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74f7487700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74f0136f30 con 0x7f74f0098670 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.744+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f74ec012630 con 0x7f74f0098670 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74e57fa700 1 --2- 172.21.15.6:0/3706217989 >> v2:172.21.15.22:6812/142603 conn(0x7f74d40357a0 0x7f74d4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f74ec047680 con 0x7f74f0098670 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74f4a22700 1 --2- 172.21.15.6:0/3706217989 >> v2:172.21.15.22:6812/142603 conn(0x7f74d40357a0 0x7f74d4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74f7487700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:02.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74f7487700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74f7487700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74f0072b00 con 0x7f74f0098670 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.745+0000 7f74f4a22700 10 monclient: get_auth_request con 0x7f74d40357a0 auth_method 0 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.747+0000 7f74f4a22700 1 --2- 172.21.15.6:0/3706217989 >> v2:172.21.15.22:6812/142603 conn(0x7f74d40357a0 0x7f74d4037c50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f74f009f9a0 tx=0x7f74e0008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.750+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f74ec011e60 con 0x7f74f0098670 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.750+0000 7f74e57fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:02.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.750+0000 7f74e57fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74f7487700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74f7487700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74f7487700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f74f009f630 con 0x7f74f0098670 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74e57fa700 1 -- 172.21.15.6:0/3706217989 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f74ec01b3c0 con 0x7f74f0098670 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74e57fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:02.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:02.997+0000 7f74e57fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:02.999 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:02.999 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:03.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 >> v2:172.21.15.22:6812/142603 conn(0x7f74d40357a0 msgr2=0x7f74d4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 --2- 172.21.15.6:0/3706217989 >> v2:172.21.15.22:6812/142603 conn(0x7f74d40357a0 0x7f74d4037c50 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f74f009f9a0 tx=0x7f74e0008040 comp rx=0 tx=0).stop 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 10 monclient: shutdown 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 msgr2=0x7f74f012f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 0x7f74f012f110 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f74ec00c600 tx=0x7f74ec007f60 comp rx=0 tx=0).stop 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 shutdown_connections 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 --2- 172.21.15.6:0/3706217989 >> v2:172.21.15.22:6812/142603 conn(0x7f74d40357a0 0x7f74d4037c50 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f74f0137b60 0x7f74f01369f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f74f0098670 0x7f74f012f110 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:03.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 --2- 172.21.15.6:0/3706217989 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f74f0132d00 0x7f74f009ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:03.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 >> 172.21.15.6:0/3706217989 conn(0x7f74f00a5480 msgr2=0x7f74f012fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:03.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 shutdown_connections 2023-09-30T00:49:03.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:03.000+0000 7f74f7487700 1 -- 172.21.15.6:0/3706217989 wait complete. 2023-09-30T00:49:06.015 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.152+0000 7f252ae50700 10 monclient: get_monmap_and_config 2023-09-30T00:49:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.152+0000 7f252ae50700 10 monclient: build_initial_monmap 2023-09-30T00:49:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.152+0000 7f252ae50700 10 monclient: monmap: 2023-09-30T00:49:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:06.153 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:06.153890+0000 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:06.153890+0000 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:06.154 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.153+0000 7f252ae50700 1 Processor -- start 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.153+0000 7f252ae50700 1 -- start start 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.153+0000 7f252ae50700 10 monclient: init 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f2524098670 0x7f2524132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): picked mon.noname-c con 0x7f2524098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 0x7f252412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:06.159 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): picked mon.noname-b con 0x7f2524098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f2524098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25240a1000 con 0x7f2524098e90 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25240a0a50 con 0x7f2524098670 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25240a0730 con 0x7f2524098a40 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f252ae50700 10 monclient(hunting): authenticate will time out at 1542.452208s 2023-09-30T00:49:06.160 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 0x7f252412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 0x7f252412ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40682/0 (socket says 172.21.15.6:40682) 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 -- 172.21.15.6:0/2917159774 learned_addr learned my addr 172.21.15.6:0/2917159774 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): get_auth_request con 0x7f2524098a40 auth_method 0 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 1 --2- 172.21.15.6:0/2917159774 >> v2:172.21.15.22:3300/0 conn(0x7f2524098670 0x7f2524132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): get_auth_request con 0x7f2524098670 auth_method 0 2023-09-30T00:49:06.161 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2528bec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient(hunting): handle_auth_done global_id 4706 payload 274 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 -- 172.21.15.6:0/2917159774 >> v2:172.21.15.22:3300/0 conn(0x7f2524098670 msgr2=0x7f2524132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 --2- 172.21.15.6:0/2917159774 >> v2:172.21.15.22:3300/0 conn(0x7f2524098670 0x7f2524132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 -- 172.21.15.6:0/2917159774 >> v1:172.21.15.22:6789/0 conn(0x7f2524098e90 legacy=0x7f252412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 monclient: found mon.noname-b 2023-09-30T00:49:06.162 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.154+0000 7f2523fff700 1 -- 172.21.15.6:0/2917159774 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2524068df0 con 0x7f2524098a40 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.156+0000 7f2523fff700 1 -- 172.21.15.6:0/2917159774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 msgr2=0x7f252412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.156+0000 7f2523fff700 1 --2- 172.21.15.6:0/2917159774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 0x7f252412ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25100099b0 tx=0x7f2510009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c452d1d921560d98 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 1 -- 172.21.15.6:0/2917159774 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2510009ea0 con 0x7f2524098a40 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: dump: 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:06.163 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 1 -- 172.21.15.6:0/2917159774 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251000ba30 con 0x7f2524098a40 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 5 monclient: authenticate success, global_id 4706 2023-09-30T00:49:06.164 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 1 -- 172.21.15.6:0/2917159774 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251000bdf0 con 0x7f2524098a40 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 4 set_mon_vals no callback set 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f25237fe700 10 monclient: dump: 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.165 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 10 monclient: shutdown 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 -- 172.21.15.6:0/2917159774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 msgr2=0x7f252412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 --2- 172.21.15.6:0/2917159774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 0x7f252412ede0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f25100099b0 tx=0x7f2510009250 comp rx=0 tx=0).stop 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 -- 172.21.15.6:0/2917159774 shutdown_connections 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 --2- 172.21.15.6:0/2917159774 >> v2:172.21.15.22:3300/0 conn(0x7f2524098670 0x7f2524132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.166 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 --2- 172.21.15.6:0/2917159774 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098a40 0x7f252412ede0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 -- 172.21.15.6:0/2917159774 >> 172.21.15.6:0/2917159774 conn(0x7f25240a5480 msgr2=0x7f25240a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 -- 172.21.15.6:0/2917159774 shutdown_connections 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.157+0000 7f252ae50700 1 -- 172.21.15.6:0/2917159774 wait complete. 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.158+0000 7f252ae50700 10 monclient: build_initial_monmap 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.158+0000 7f252ae50700 10 monclient: monmap: 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:06.167 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:06.168 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:06.168 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:06.168 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:06.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.158+0000 7f252ae50700 1 Processor -- start 2023-09-30T00:49:06.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.158+0000 7f252ae50700 1 -- start start 2023-09-30T00:49:06.168 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.158+0000 7f252ae50700 10 monclient: init 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 0x7f2524141b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): picked mon.noname-c con 0x7f2524098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2524098a40 0x7f252412f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): picked mon.noname-b con 0x7f2524098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 0x7f252409c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): picked mon.noname-a con 0x7f2524098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:06.172 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2524135390 con 0x7f2524098e90 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f252412da70 con 0x7f2524098a40 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25241429e0 con 0x7f2524098670 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f252ae50700 10 monclient(hunting): authenticate will time out at 1542.457429s 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 0x7f2524141b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 0x7f2524141b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40688/0 (socket says 172.21.15.6:40688) 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 1 -- 172.21.15.6:0/3422188037 learned_addr learned my addr 172.21.15.6:0/3422188037 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:06.173 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): get_auth_request con 0x7f2524098670 auth_method 0 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 0x7f252409c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): get_auth_request con 0x7f2524098e90 auth_method 0 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:06.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2523fff700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2524098a40 0x7f252412f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2523fff700 10 monclient(hunting): get_auth_request con 0x7f2524098a40 auth_method 0 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2523fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2523fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2523fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient(hunting): handle_auth_done global_id 4686 payload 824 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 -- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 msgr2=0x7f2524141b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 0x7f2524141b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 -- 172.21.15.6:0/3422188037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2524098a40 msgr2=0x7f252412f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2524098a40 0x7f252412f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient: _finish_hunting 0 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 monclient: found mon.noname-a 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f25293ed700 1 -- 172.21.15.6:0/3422188037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25100094f0 con 0x7f2524098e90 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.159+0000 7f2528bec700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 0x7f2524141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f25293ed700 1 -- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 msgr2=0x7f252409c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f25293ed700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 0x7f252409c7c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f25240a0650 tx=0x7f251c002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c012840 con 0x7f2524098e90 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:06.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: dump: 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: _finish_auth 0 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:06.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f251c0129c0 con 0x7f2524098e90 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 5 monclient: authenticate success, global_id 4686 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 10 monclient: _renew_subs 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f252409f010 con 0x7f2524098e90 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f251c012d80 con 0x7f2524098e90 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2520ff9700 10 monclient: dump: 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:06.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f2522ffd700 4 set_mon_vals no callback set 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 10 monclient: _renew_subs 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.162+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f252409e370 con 0x7f2524098e90 2023-09-30T00:49:06.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f252ae50700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f252ae50700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f252ae50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2524072b00 con 0x7f2524098e90 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f251c011590 con 0x7f2524098e90 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f2520ff9700 1 --2- 172.21.15.6:0/3422188037 >> v2:172.21.15.22:6812/142603 conn(0x7f25080357a0 0x7f2508037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f251c047c50 con 0x7f2524098e90 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f2528bec700 1 --2- 172.21.15.6:0/3422188037 >> v2:172.21.15.22:6812/142603 conn(0x7f25080357a0 0x7f2508037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.163+0000 7f2528bec700 10 monclient: get_auth_request con 0x7f25080357a0 auth_method 0 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.166+0000 7f2528bec700 1 --2- 172.21.15.6:0/3422188037 >> v2:172.21.15.22:6812/142603 conn(0x7f25080357a0 0x7f2508037c50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2518009fd0 tx=0x7f25180091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.168+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f251c0108d0 con 0x7f2524098e90 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.168+0000 7f2520ff9700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:06.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.168+0000 7f2520ff9700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:06.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f252ae50700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:06.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f252ae50700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:06.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f252ae50700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:06.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2524135150 con 0x7f2524098e90 2023-09-30T00:49:06.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f2520ff9700 1 -- 172.21.15.6:0/3422188037 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f251c016030 con 0x7f2524098e90 2023-09-30T00:49:06.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f2520ff9700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:06.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.415+0000 7f2520ff9700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:06.417 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:06.417 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 >> v2:172.21.15.22:6812/142603 conn(0x7f25080357a0 msgr2=0x7f2508037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 --2- 172.21.15.6:0/3422188037 >> v2:172.21.15.22:6812/142603 conn(0x7f25080357a0 0x7f2508037c50 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f2518009fd0 tx=0x7f25180091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 10 monclient: shutdown 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 msgr2=0x7f252409c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 0x7f252409c7c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f25240a0650 tx=0x7f251c002620 comp rx=0 tx=0).stop 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 shutdown_connections 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 --2- 172.21.15.6:0/3422188037 >> v2:172.21.15.22:6812/142603 conn(0x7f25080357a0 0x7f2508037c50 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.419 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2524098e90 0x7f252409c7c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2524098670 0x7f2524141b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 --2- 172.21.15.6:0/3422188037 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2524098a40 0x7f252412f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:06.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 >> 172.21.15.6:0/3422188037 conn(0x7f25240a5480 msgr2=0x7f2524069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:06.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 shutdown_connections 2023-09-30T00:49:06.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:06.418+0000 7f252ae50700 1 -- 172.21.15.6:0/3422188037 wait complete. 2023-09-30T00:49:09.432 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:09.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.569+0000 7f5866113700 10 monclient: get_monmap_and_config 2023-09-30T00:49:09.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.569+0000 7f5866113700 10 monclient: build_initial_monmap 2023-09-30T00:49:09.570 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.569+0000 7f5866113700 10 monclient: monmap: 2023-09-30T00:49:09.570 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:09.570 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:09.570 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:09.570643+0000 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:09.570643+0000 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:09.571 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 Processor -- start 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 -- start start 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient: init 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient: _add_conns ranks=[1,2,4,3,0] 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f5860098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:09.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f5860137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): picked mon.noname-b con 0x7f5860132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5860137b60 0x7f5860139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): picked mon.noname-c con 0x7f5860137b60 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58600a1000 con 0x7f5860098670 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5860099150 con 0x7f5860137b60 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58600a0a50 con 0x7f5860132d00 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f5866113700 10 monclient(hunting): authenticate will time out at 1545.868814s 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f585effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f5860137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:09.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.570+0000 7f585effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f5860137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:49626/0 (socket says 172.21.15.6:49626) 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 1 -- 172.21.15.6:0/1175714269 learned_addr learned my addr 172.21.15.6:0/1175714269 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): get_auth_request con 0x7f5860132d00 auth_method 0 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 1 --2- 172.21.15.6:0/1175714269 >> v2:172.21.15.22:3300/0 conn(0x7f5860137b60 0x7f5860139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): get_auth_request con 0x7f5860137b60 auth_method 0 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:09.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585ffff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient(hunting): handle_auth_done global_id 4712 payload 274 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 1 -- 172.21.15.6:0/1175714269 >> v2:172.21.15.22:3300/0 conn(0x7f5860137b60 msgr2=0x7f5860139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 1 --2- 172.21.15.6:0/1175714269 >> v2:172.21.15.22:3300/0 conn(0x7f5860137b60 0x7f5860139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 1 -- 172.21.15.6:0/1175714269 >> v1:172.21.15.22:6789/0 conn(0x7f5860098670 legacy=0x7f5860098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 1 monclient: found mon.noname-b 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:09.579 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.571+0000 7f585effd700 1 -- 172.21.15.6:0/1175714269 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58600a0730 con 0x7f5860132d00 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585effd700 1 -- 172.21.15.6:0/1175714269 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 msgr2=0x7f5860137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585effd700 1 --2- 172.21.15.6:0/1175714269 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f5860137620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f58480099b0 tx=0x7f5848009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97c2aabe7bdc94b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 1 -- 172.21.15.6:0/1175714269 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848009ea0 con 0x7f5860132d00 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: dump: 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.580 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 1 -- 172.21.15.6:0/1175714269 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f584800ba30 con 0x7f5860132d00 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f5866113700 5 monclient: authenticate success, global_id 4712 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f5866113700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:09.581 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 1 -- 172.21.15.6:0/1175714269 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f584800bdf0 con 0x7f5860132d00 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f5866113700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f5866113700 4 set_mon_vals no callback set 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.573+0000 7f585e7fc700 10 monclient: dump: 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:09.582 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 10 monclient: shutdown 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 -- 172.21.15.6:0/1175714269 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 msgr2=0x7f5860137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 --2- 172.21.15.6:0/1175714269 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f5860137620 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f58480099b0 tx=0x7f5848009250 comp rx=0 tx=0).stop 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 -- 172.21.15.6:0/1175714269 shutdown_connections 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 --2- 172.21.15.6:0/1175714269 >> v2:172.21.15.22:3300/0 conn(0x7f5860137b60 0x7f5860139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 --2- 172.21.15.6:0/1175714269 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f5860137620 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 -- 172.21.15.6:0/1175714269 >> 172.21.15.6:0/1175714269 conn(0x7f58600a5480 msgr2=0x7f58600a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:09.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 -- 172.21.15.6:0/1175714269 shutdown_connections 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.574+0000 7f5866113700 1 -- 172.21.15.6:0/1175714269 wait complete. 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.575+0000 7f5866113700 10 monclient: build_initial_monmap 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.575+0000 7f5866113700 10 monclient: monmap: 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:09.584 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:09.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.575+0000 7f5866113700 1 Processor -- start 2023-09-30T00:49:09.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.575+0000 7f5866113700 1 -- start start 2023-09-30T00:49:09.585 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.575+0000 7f5866113700 10 monclient: init 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5860098670 0x7f586009c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): picked mon.noname-a con 0x7f5860098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f586009cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): picked mon.noname-c con 0x7f5860132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:09.589 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5860137b60 0x7f586009d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): picked mon.noname-b con 0x7f5860137b60 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58600a1500 con 0x7f5860098670 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5860137880 con 0x7f5860137b60 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5860133410 con 0x7f5860132d00 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f5866113700 10 monclient(hunting): authenticate will time out at 1545.874275s 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5860098670 0x7f586009c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:09.590 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f586009cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5860098670 0x7f586009c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53288/0 (socket says 172.21.15.6:53288) 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f586009cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:49636/0 (socket says 172.21.15.6:49636) 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 1 -- 172.21.15.6:0/1129216319 learned_addr learned my addr 172.21.15.6:0/1129216319 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): get_auth_request con 0x7f5860098670 auth_method 0 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): get_auth_request con 0x7f5860132d00 auth_method 0 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:09.591 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585ffff700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5860137b60 0x7f586009d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585ffff700 10 monclient(hunting): get_auth_request con 0x7f5860137b60 auth_method 0 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585ffff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585ffff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585ffff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient(hunting): handle_auth_done global_id 4715 payload 824 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 -- 172.21.15.6:0/1129216319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5860137b60 msgr2=0x7f586009d220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5860137b60 0x7f586009d220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.592 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 -- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5860098670 msgr2=0x7f586009c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5860098670 0x7f586009c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 monclient: found mon.noname-c 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585effd700 1 -- 172.21.15.6:0/1129216319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58480094f0 con 0x7f5860132d00 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.576+0000 7f585ffff700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5860137b60 0x7f586009d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f585effd700 1 -- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 msgr2=0x7f586009cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f585effd700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f586009cce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f586009fe60 tx=0x7f5848016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848006a90 con 0x7f5860132d00 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:09.593 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: dump: 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:09.594 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5848020070 con 0x7f5860132d00 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5848011d60 con 0x7f5860132d00 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 5 monclient: authenticate success, global_id 4715 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f58577fe700 10 monclient: dump: 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:09.595 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f585dffb700 4 set_mon_vals no callback set 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 10 monclient: _renew_subs 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58601336b0 con 0x7f5860132d00 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 10 monclient: _renew_subs 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.579+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f586009dc40 con 0x7f5860132d00 2023-09-30T00:49:09.596 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5848022030 con 0x7f5860132d00 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f5866113700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f5866113700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f5866113700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f58577fe700 1 --2- 172.21.15.6:0/1129216319 >> v2:172.21.15.22:6812/142603 conn(0x7f58400357a0 0x7f5840037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f586009f7c0 con 0x7f5860132d00 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5848049500 con 0x7f5860132d00 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f585f7fe700 1 --2- 172.21.15.6:0/1129216319 >> v2:172.21.15.22:6812/142603 conn(0x7f58400357a0 0x7f5840037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.580+0000 7f585f7fe700 10 monclient: get_auth_request con 0x7f58400357a0 auth_method 0 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.583+0000 7f585f7fe700 1 --2- 172.21.15.6:0/1129216319 >> v2:172.21.15.22:6812/142603 conn(0x7f58400357a0 0x7f5840037c50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5850007fd0 tx=0x7f5850007f60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.585+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58480249c0 con 0x7f5860132d00 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.585+0000 7f58577fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:09.597 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.585+0000 7f58577fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:09.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f5866113700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:09.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f5866113700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:09.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f5866113700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f586009f7c0 con 0x7f5860132d00 2023-09-30T00:49:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f58577fe700 1 -- 172.21.15.6:0/1129216319 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f58480243d0 con 0x7f5860132d00 2023-09-30T00:49:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f58577fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:09.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.834+0000 7f58577fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:09.836 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:09.836 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 >> v2:172.21.15.22:6812/142603 conn(0x7f58400357a0 msgr2=0x7f5840037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 --2- 172.21.15.6:0/1129216319 >> v2:172.21.15.22:6812/142603 conn(0x7f58400357a0 0x7f5840037c50 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f5850007fd0 tx=0x7f5850007f60 comp rx=0 tx=0).stop 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 10 monclient: shutdown 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 msgr2=0x7f586009cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f586009cce0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f586009fe60 tx=0x7f5848016000 comp rx=0 tx=0).stop 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 shutdown_connections 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 --2- 172.21.15.6:0/1129216319 >> v2:172.21.15.22:6812/142603 conn(0x7f58400357a0 0x7f5840037c50 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5860137b60 0x7f586009d220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5860098670 0x7f586009c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 --2- 172.21.15.6:0/1129216319 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5860132d00 0x7f586009cce0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:09.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 >> 172.21.15.6:0/1129216319 conn(0x7f58600a5480 msgr2=0x7f5860068ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:09.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 shutdown_connections 2023-09-30T00:49:09.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:09.837+0000 7f5866113700 1 -- 172.21.15.6:0/1129216319 wait complete. 2023-09-30T00:49:12.852 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:12.990 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.990+0000 7fd185665700 10 monclient: get_monmap_and_config 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.990+0000 7fd185665700 10 monclient: build_initial_monmap 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.990+0000 7fd185665700 10 monclient: monmap: 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:12.991508+0000 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:12.991508+0000 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:12.991 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:12.992 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:12.992 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:12.992 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:12.992 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:12.992 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 Processor -- start 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 -- start start 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient: init 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient: _add_conns ranks=[3,2,1,4,0] 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1800631f0 0x7fd18009adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): picked mon.noname-a con 0x7fd1800631f0 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 0x7fd18009b700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): picked mon.noname-b con 0x7fd18009b330 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:12.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd180132270 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1800a12f0 con 0x7fd180132270 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd18009bc40 con 0x7fd1800631f0 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800635c0 con 0x7fd18009b330 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd185665700 10 monclient(hunting): authenticate will time out at 1549.289676s 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1800631f0 0x7fd18009adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd177fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 0x7fd18009b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7fd1800631f0 0x7fd18009adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53302/0 (socket says 172.21.15.6:53302) 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd177fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 0x7fd18009b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:49646/0 (socket says 172.21.15.6:49646) 2023-09-30T00:49:13.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 1 -- 172.21.15.6:0/588941167 learned_addr learned my addr 172.21.15.6:0/588941167 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 10 monclient(hunting): get_auth_request con 0x7fd1800631f0 auth_method 0 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd177fff700 10 monclient(hunting): get_auth_request con 0x7fd18009b330 auth_method 0 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd177fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd177fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd177fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.991+0000 7fd17effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:13.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd177fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd177fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd177fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 10 monclient(hunting): handle_auth_done global_id 4692 payload 274 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 1 -- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 msgr2=0x7fd18009b700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 1 --2- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 0x7fd18009b700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 1 -- 172.21.15.6:0/588941167 >> v1:172.21.15.22:6789/0 conn(0x7fd180132270 legacy=0x7fd180132710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 1 monclient: found mon.noname-a 2023-09-30T00:49:13.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd17effd700 1 -- 172.21.15.6:0/588941167 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1800a09f0 con 0x7fd1800631f0 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.992+0000 7fd177fff700 1 --2- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 0x7fd18009b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17effd700 1 -- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1800631f0 msgr2=0x7fd18009adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17effd700 1 --2- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1800631f0 0x7fd18009adf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd1700099b0 tx=0x7fd170009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92e9c4a8173732cd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 1 -- 172.21.15.6:0/588941167 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd170009ea0 con 0x7fd1800631f0 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: dump: 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:13.003 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 1 -- 172.21.15.6:0/588941167 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd17000ba30 con 0x7fd1800631f0 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd185665700 5 monclient: authenticate success, global_id 4692 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd185665700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:13.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 1 -- 172.21.15.6:0/588941167 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd17000bdf0 con 0x7fd1800631f0 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd185665700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd185665700 4 set_mon_vals no callback set 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.994+0000 7fd17e7fc700 10 monclient: dump: 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:13.005 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 10 monclient: shutdown 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 -- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1800631f0 msgr2=0x7fd18009adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 --2- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1800631f0 0x7fd18009adf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fd1700099b0 tx=0x7fd170009250 comp rx=0 tx=0).stop 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 -- 172.21.15.6:0/588941167 shutdown_connections 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 --2- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd1800631f0 0x7fd18009adf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 --2- 172.21.15.6:0/588941167 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd18009b330 0x7fd18009b700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 -- 172.21.15.6:0/588941167 >> 172.21.15.6:0/588941167 conn(0x7fd1800680c0 msgr2=0x7fd18009e450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 -- 172.21.15.6:0/588941167 shutdown_connections 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 1 -- 172.21.15.6:0/588941167 wait complete. 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 10 monclient: build_initial_monmap 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.995+0000 7fd185665700 10 monclient: monmap: 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:13.007 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 Processor -- start 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 -- start start 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient: init 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd1800631f0 0x7fd180141980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): picked mon.noname-c con 0x7fd1800631f0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd18009b330 0x7fd180141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): picked mon.noname-b con 0x7fd18009b330 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 0x7fd18011ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:13.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): picked mon.noname-a con 0x7fd180132270 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800a0bd0 con 0x7fd180132270 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1800a2420 con 0x7fd18009b330 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd180122670 con 0x7fd1800631f0 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd185665700 10 monclient(hunting): authenticate will time out at 1549.294782s 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd1800631f0 0x7fd180141980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 0x7fd18011ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd1800631f0 0x7fd180141980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:49662/0 (socket says 172.21.15.6:49662) 2023-09-30T00:49:13.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17f7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 0x7fd18011ecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:53314/0 (socket says 172.21.15.6:53314) 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 1 -- 172.21.15.6:0/2004236461 learned_addr learned my addr 172.21.15.6:0/2004236461 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 10 monclient(hunting): get_auth_request con 0x7fd1800631f0 auth_method 0 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17effd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17f7fe700 10 monclient(hunting): get_auth_request con 0x7fd180132270 auth_method 0 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17f7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17f7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.996+0000 7fd17f7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:13.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17effd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17effd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd18009b330 0x7fd180141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17effd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): get_auth_request con 0x7fd18009b330 auth_method 0 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd177fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 10 monclient(hunting): handle_auth_done global_id 4695 payload 824 2023-09-30T00:49:13.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 1 -- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd1800631f0 msgr2=0x7fd180141980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd1800631f0 0x7fd180141980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 1 -- 172.21.15.6:0/2004236461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd18009b330 msgr2=0x7fd180141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd18009b330 0x7fd180141ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 1 monclient: found mon.noname-a 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:13.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.997+0000 7fd17f7fe700 1 -- 172.21.15.6:0/2004236461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd168009420 con 0x7fd180132270 2023-09-30T00:49:13.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd17f7fe700 1 -- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 msgr2=0x7fd18011ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:13.014 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd17f7fe700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 0x7fd18011ecc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd17800c810 tx=0x7fd178002620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178012840 con 0x7fd180132270 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: dump: 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:13.015 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd1780129c0 con 0x7fd180132270 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 5 monclient: authenticate success, global_id 4695 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 10 monclient: _renew_subs 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1700094f0 con 0x7fd180132270 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd178012d80 con 0x7fd180132270 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:13.016 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd176ffd700 10 monclient: dump: 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd17dffb700 4 set_mon_vals no callback set 2023-09-30T00:49:13.017 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 10 monclient: _renew_subs 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:12.999+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd180120860 con 0x7fd180132270 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd178011590 con 0x7fd180132270 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd185665700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd185665700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd185665700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1801228b0 con 0x7fd180132270 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd176ffd700 1 --2- 172.21.15.6:0/2004236461 >> v2:172.21.15.22:6812/142603 conn(0x7fd1600357f0 0x7fd160037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.000+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd1780474d0 con 0x7fd180132270 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.001+0000 7fd17effd700 1 --2- 172.21.15.6:0/2004236461 >> v2:172.21.15.22:6812/142603 conn(0x7fd1600357f0 0x7fd160037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.001+0000 7fd17effd700 10 monclient: get_auth_request con 0x7fd1600357f0 auth_method 0 2023-09-30T00:49:13.018 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.003+0000 7fd17effd700 1 --2- 172.21.15.6:0/2004236461 >> v2:172.21.15.22:6812/142603 conn(0x7fd1600357f0 0x7fd160037ca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd1700030a0 tx=0x7fd170016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.005+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd1801228b0 con 0x7fd180132270 2023-09-30T00:49:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.005+0000 7fd176ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:13.019 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.005+0000 7fd176ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd185665700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd185665700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd185665700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd1801228b0 con 0x7fd180132270 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd176ffd700 1 -- 172.21.15.6:0/2004236461 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fd1801228b0 con 0x7fd180132270 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd176ffd700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:13.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.258+0000 7fd176ffd700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:13.260 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:13.260 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 >> v2:172.21.15.22:6812/142603 conn(0x7fd1600357f0 msgr2=0x7fd160037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 --2- 172.21.15.6:0/2004236461 >> v2:172.21.15.22:6812/142603 conn(0x7fd1600357f0 0x7fd160037ca0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fd1700030a0 tx=0x7fd170016000 comp rx=0 tx=0).stop 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 10 monclient: shutdown 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 msgr2=0x7fd18011ecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 0x7fd18011ecc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fd17800c810 tx=0x7fd178002620 comp rx=0 tx=0).stop 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 shutdown_connections 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 --2- 172.21.15.6:0/2004236461 >> v2:172.21.15.22:6812/142603 conn(0x7fd1600357f0 0x7fd160037ca0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd180132270 0x7fd18011ecc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd1800631f0 0x7fd180141980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 --2- 172.21.15.6:0/2004236461 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd18009b330 0x7fd180141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:13.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 >> 172.21.15.6:0/2004236461 conn(0x7fd1800680c0 msgr2=0x7fd18009dae0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:13.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 shutdown_connections 2023-09-30T00:49:13.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:13.261+0000 7fd185665700 1 -- 172.21.15.6:0/2004236461 wait complete. 2023-09-30T00:49:13.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.339+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.339+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=10,kv=8,logm=72} crc {auth=795114675,config=3140514009,health=3800711315,kv=1804324294,logm=3827332481}) 2023-09-30T00:49:13.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.339+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=9,config=2,health=12,kv=8,logm=69} crc {auth=954757949,config=3140514009,health=1307886691,kv=1804324294,logm=625974677}) 2023-09-30T00:49:13.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.339+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.339+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=8,config=2,health=10,kv=8,logm=72} crc {auth=795114675,config=3140514009,health=3800711315,kv=1804324294,logm=3827332481}) 2023-09-30T00:49:13.340 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.339+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=9,config=2,health=12,kv=8,logm=69} crc {auth=3266410041,config=3140514009,health=1307886691,kv=1804324294,logm=3422993559}) 2023-09-30T00:49:13.388 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.388 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448457710}) 2023-09-30T00:49:13.388 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3551536801}) 2023-09-30T00:49:13.388 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448457710}) 2023-09-30T00:49:13.389 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394790328}) 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972563359}) 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742447187}) 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2972563359}) 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.388+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537708385}) 2023-09-30T00:49:13.390 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.389+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.389+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374269135}) 2023-09-30T00:49:13.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.389+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1763049202}) 2023-09-30T00:49:13.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.389+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.389+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3374269135}) 2023-09-30T00:49:13.391 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.389+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2371181950}) 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.390+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.390+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151727519}) 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.390+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3958502262}) 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.390+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.390+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=151727519}) 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.390+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=657755924}) 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.391+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.392 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.391+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980404820}) 2023-09-30T00:49:13.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.391+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2997739960}) 2023-09-30T00:49:13.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.391+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.391+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2980404820}) 2023-09-30T00:49:13.393 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.391+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=636458450}) 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osdmap=21} crc {logm=4218423299,mdsmap=1531151339,mgr=2406852585,mgr_command_descs=3077679017,mgr_metadata=3025485890,mgrstat=1836322571,mon_config_key=476770126,monmap=898192722,osd_metadata=2288311013,osdmap=1833056088}) 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=46,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=5,monmap=3,osd_metadata=6,osd_pg_creating=1,osdmap=18} crc {logm=2237120464,mdsmap=1531151339,mgr=2406852585,mgr_command_descs=3077679017,mgr_metadata=3025485890,mgrstat=2592474008,mon_config_key=476770126,monmap=898192723,osd_metadata=1082820484,osd_pg_creating=2927750135,osdmap=1066078576}) 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=45,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osdmap=21} crc {logm=4218423299,mdsmap=1531151339,mgr=2406852585,mgr_command_descs=3077679017,mgr_metadata=3025485890,mgrstat=1836322571,mon_config_key=476770126,monmap=898192722,osd_metadata=2288311013,osdmap=1833056088}) 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=45,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=7,mon_config_key=5,monmap=3,osd_metadata=6,osdmap=22} crc {logm=2455531826,mdsmap=3585761405,mgr=2406852585,mgr_command_descs=3077679017,mgr_metadata=3025485890,mgrstat=567093636,mon_config_key=476770126,monmap=898192722,osd_metadata=1082820484,osdmap=421751010}) 2023-09-30T00:49:13.394 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.392+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.393+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=671463125}) 2023-09-30T00:49:13.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.393+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=14} crc {osdmap=3793515232}) 2023-09-30T00:49:13.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.393+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : scrub mismatch 2023-09-30T00:49:13.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.393+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=14} crc {osdmap=671463125}) 2023-09-30T00:49:13.395 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:13.393+0000 7f825d8da700 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=339629226}) 2023-09-30T00:49:16.276 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:16.415 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.415+0000 7f661b330700 10 monclient: get_monmap_and_config 2023-09-30T00:49:16.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.415+0000 7f661b330700 10 monclient: build_initial_monmap 2023-09-30T00:49:16.416 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.415+0000 7f661b330700 10 monclient: monmap: 2023-09-30T00:49:16.416 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:16.416 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:16.416 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:16.416299+0000 2023-09-30T00:49:16.416 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:16.416299+0000 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:16.417 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.415+0000 7f661b330700 1 Processor -- start 2023-09-30T00:49:16.420 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.415+0000 7f661b330700 1 -- start start 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.415+0000 7f661b330700 10 monclient: init 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f6614098670 0x7f6614132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): picked mon.noname-c con 0x7f6614098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 0x7f661412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): picked mon.noname-b con 0x7f6614098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f6614098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66140a1000 con 0x7f6614098e90 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66140a0a50 con 0x7f6614098670 2023-09-30T00:49:16.421 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66140a0730 con 0x7f6614098a40 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f661b330700 10 monclient(hunting): authenticate will time out at 1552.714495s 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 0x7f661412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66198cd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f6614098e90 0x7f661412f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41568/0 (socket says 172.21.15.6:41568) 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 0x7f661412ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:60476/0 (socket says 172.21.15.6:60476) 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66198cd700 1 -- 172.21.15.6:0/1351632264 learned_addr learned my addr 172.21.15.6:0/1351632264 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): get_auth_request con 0x7f6614098a40 auth_method 0 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:16.422 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66190cc700 1 --2- 172.21.15.6:0/1351632264 >> v2:172.21.15.22:3300/0 conn(0x7f6614098670 0x7f6614132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66190cc700 10 monclient(hunting): get_auth_request con 0x7f6614098670 auth_method 0 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66190cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66190cc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66190cc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient(hunting): handle_auth_done global_id 4724 payload 274 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 1 -- 172.21.15.6:0/1351632264 >> v2:172.21.15.22:3300/0 conn(0x7f6614098670 msgr2=0x7f6614132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 1 --2- 172.21.15.6:0/1351632264 >> v2:172.21.15.22:3300/0 conn(0x7f6614098670 0x7f6614132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 1 -- 172.21.15.6:0/1351632264 >> v1:172.21.15.6:6789/0 conn(0x7f6614098e90 legacy=0x7f661412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:16.423 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient: _finish_hunting 0 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 1 monclient: found mon.noname-b 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.416+0000 7f66188cb700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.417+0000 7f66188cb700 1 -- 172.21.15.6:0/1351632264 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6614068df0 con 0x7f6614098a40 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.417+0000 7f66190cc700 1 --2- 172.21.15.6:0/1351632264 >> v2:172.21.15.22:3300/0 conn(0x7f6614098670 0x7f6614132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f66188cb700 1 -- 172.21.15.6:0/1351632264 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 msgr2=0x7f661412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f66188cb700 1 --2- 172.21.15.6:0/1351632264 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 0x7f661412ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66040098e0 tx=0x7f66040091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbb3311b73f0122c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 1 -- 172.21.15.6:0/1351632264 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6604009e80 con 0x7f6614098a40 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: dump: 2023-09-30T00:49:16.424 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: _finish_auth 0 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 1 -- 172.21.15.6:0/1351632264 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f660400ba30 con 0x7f6614098a40 2023-09-30T00:49:16.425 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 5 monclient: authenticate success, global_id 4724 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 4 set_mon_vals no callback set 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 1 -- 172.21.15.6:0/1351632264 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f660400bdb0 con 0x7f6614098a40 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f660bfff700 10 monclient: dump: 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.426 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 10 monclient: shutdown 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 1 -- 172.21.15.6:0/1351632264 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 msgr2=0x7f661412ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.419+0000 7f661b330700 1 --2- 172.21.15.6:0/1351632264 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 0x7f661412ede0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f66040098e0 tx=0x7f66040091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 1 -- 172.21.15.6:0/1351632264 shutdown_connections 2023-09-30T00:49:16.427 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 1 --2- 172.21.15.6:0/1351632264 >> v2:172.21.15.22:3300/0 conn(0x7f6614098670 0x7f6614132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 1 --2- 172.21.15.6:0/1351632264 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098a40 0x7f661412ede0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 1 -- 172.21.15.6:0/1351632264 >> 172.21.15.6:0/1351632264 conn(0x7f66140a5480 msgr2=0x7f66140a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:16.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 1 -- 172.21.15.6:0/1351632264 shutdown_connections 2023-09-30T00:49:16.428 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 1 -- 172.21.15.6:0/1351632264 wait complete. 2023-09-30T00:49:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 10 monclient: build_initial_monmap 2023-09-30T00:49:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.420+0000 7f661b330700 10 monclient: monmap: 2023-09-30T00:49:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:16.432 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.421+0000 7f661b330700 1 Processor -- start 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.421+0000 7f661b330700 1 -- start start 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.421+0000 7f661b330700 10 monclient: init 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:49:16.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 0x7f6614131710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): picked mon.noname-a con 0x7f6614098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6614098a40 0x7f66141417d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): picked mon.noname-b con 0x7f6614098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098e90 0x7f6614141d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): picked mon.noname-c con 0x7f6614098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6614130d50 con 0x7f6614098670 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661412f8a0 con 0x7f6614098a40 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6614130790 con 0x7f6614098e90 2023-09-30T00:49:16.434 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f661b330700 10 monclient(hunting): authenticate will time out at 1552.720471s 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 0x7f6614131710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 0x7f6614131710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50860/0 (socket says 172.21.15.6:50860) 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098e90 0x7f6614141d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 -- 172.21.15.6:0/1200583220 learned_addr learned my addr 172.21.15.6:0/1200583220 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): get_auth_request con 0x7f6614098670 auth_method 0 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): get_auth_request con 0x7f6614098e90 auth_method 0 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:16.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6614098a40 0x7f66141417d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): get_auth_request con 0x7f6614098a40 auth_method 0 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:16.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66188cb700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient(hunting): handle_auth_done global_id 4698 payload 824 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 -- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098e90 msgr2=0x7f6614141d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098e90 0x7f6614141d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 -- 172.21.15.6:0/1200583220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6614098a40 msgr2=0x7f66141417d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6614098a40 0x7f66141417d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient: _finish_hunting 0 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 monclient: found mon.noname-a 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:16.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66190cc700 1 -- 172.21.15.6:0/1200583220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6604009420 con 0x7f6614098670 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.422+0000 7f66198cd700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098e90 0x7f6614141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66190cc700 1 -- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 msgr2=0x7f6614131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66190cc700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 0x7f6614131710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f661000e8d0 tx=0x7f6610008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66100093b0 con 0x7f6614098670 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: dump: 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.438 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: _finish_auth 0 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6610013620 con 0x7f6614098670 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 5 monclient: authenticate success, global_id 4698 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:16.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66100139c0 con 0x7f6614098670 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f660b7fe700 4 set_mon_vals no callback set 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f66097fa700 10 monclient: dump: 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:16.440 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 10 monclient: _renew_subs 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6614130a70 con 0x7f6614098670 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 10 monclient: _renew_subs 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.425+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6614142250 con 0x7f6614098670 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6610012370 con 0x7f6614098670 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f661b330700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f661b330700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:16.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f661b330700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6614072b00 con 0x7f6614098670 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f66097fa700 1 --2- 172.21.15.6:0/1200583220 >> v2:172.21.15.22:6812/142603 conn(0x7f65f80357a0 0x7f65f8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.426+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6610047630 con 0x7f6614098670 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.427+0000 7f66188cb700 1 --2- 172.21.15.6:0/1200583220 >> v2:172.21.15.22:6812/142603 conn(0x7f65f80357a0 0x7f65f8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.427+0000 7f66188cb700 10 monclient: get_auth_request con 0x7f65f80357a0 auth_method 0 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.429+0000 7f66188cb700 1 --2- 172.21.15.6:0/1200583220 >> v2:172.21.15.22:6812/142603 conn(0x7f65f80357a0 0x7f65f8037c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f66040030c0 tx=0x7f6604016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.431+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f661000f040 con 0x7f6614098670 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.431+0000 7f66097fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:16.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.431+0000 7f66097fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f661b330700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f661b330700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f661b330700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6614063920 con 0x7f6614098670 2023-09-30T00:49:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f66097fa700 1 -- 172.21.15.6:0/1200583220 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f6610020020 con 0x7f6614098670 2023-09-30T00:49:16.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f66097fa700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:16.681 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.679+0000 7f66097fa700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:16.681 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:16.681 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:16.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.681+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 >> v2:172.21.15.22:6812/142603 conn(0x7f65f80357a0 msgr2=0x7f65f8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.681+0000 7f661b330700 1 --2- 172.21.15.6:0/1200583220 >> v2:172.21.15.22:6812/142603 conn(0x7f65f80357a0 0x7f65f8037c50 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f66040030c0 tx=0x7f6604016000 comp rx=0 tx=0).stop 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.681+0000 7f661b330700 10 monclient: shutdown 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.681+0000 7f661b330700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.681+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 msgr2=0x7f6614131710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.681+0000 7f661b330700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 0x7f6614131710 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f661000e8d0 tx=0x7f6610008040 comp rx=0 tx=0).stop 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 shutdown_connections 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 --2- 172.21.15.6:0/1200583220 >> v2:172.21.15.22:6812/142603 conn(0x7f65f80357a0 0x7f65f8037c50 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6614098e90 0x7f6614141d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6614098670 0x7f6614131710 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 --2- 172.21.15.6:0/1200583220 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6614098a40 0x7f66141417d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 >> 172.21.15.6:0/1200583220 conn(0x7f66140a5480 msgr2=0x7f6614133aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:16.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 shutdown_connections 2023-09-30T00:49:16.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:16.682+0000 7f661b330700 1 -- 172.21.15.6:0/1200583220 wait complete. 2023-09-30T00:49:19.696 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.833+0000 7fdaa7f8c700 10 monclient: get_monmap_and_config 2023-09-30T00:49:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.833+0000 7fdaa7f8c700 10 monclient: build_initial_monmap 2023-09-30T00:49:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.833+0000 7fdaa7f8c700 10 monclient: monmap: 2023-09-30T00:49:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:19.834 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:19.834616+0000 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:19.834616+0000 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:19.835 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.833+0000 7fdaa7f8c700 1 Processor -- start 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 1 -- start start 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient: init 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient: _add_conns ranks=[0,2,1,4,3] 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fdaa0098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 0x7fdaa0137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient(hunting): picked mon.noname-b con 0x7fdaa0132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fdaa0137b60 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa00a1000 con 0x7fdaa0098670 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaa00a0730 con 0x7fdaa0137b60 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa0099150 con 0x7fdaa0132d00 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa7f8c700 10 monclient(hunting): authenticate will time out at 1556.132886s 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa5d28700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fdaa0098670 0x7fdaa0098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41574/0 (socket says 172.21.15.6:41574) 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa5527700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 0x7fdaa0137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.834+0000 7fdaa5d28700 1 -- 172.21.15.6:0/1584075381 learned_addr learned my addr 172.21.15.6:0/1584075381 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:19.838 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): get_auth_request con 0x7fdaa0132d00 auth_method 0 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient(hunting): handle_auth_done global_id 4730 payload 274 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 1 -- 172.21.15.6:0/1584075381 >> v1:172.21.15.22:6789/0 conn(0x7fdaa0137b60 legacy=0x7fdaa0139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 1 -- 172.21.15.6:0/1584075381 >> v1:172.21.15.6:6789/0 conn(0x7fdaa0098670 legacy=0x7fdaa0098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient: _finish_hunting 0 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 1 monclient: found mon.noname-b 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:19.839 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:19.840 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.835+0000 7fdaa5527700 1 -- 172.21.15.6:0/1584075381 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaa0068df0 con 0x7fdaa0132d00 2023-09-30T00:49:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa5527700 1 -- 172.21.15.6:0/1584075381 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 msgr2=0x7fdaa0137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa5527700 1 --2- 172.21.15.6:0/1584075381 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 0x7fdaa0137620 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda90009910 tx=0x7fda900091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0acfbb3f6d154fb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 1 -- 172.21.15.6:0/1584075381 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda9000b9f0 con 0x7fdaa0132d00 2023-09-30T00:49:19.845 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: dump: 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:19.846 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: _finish_auth 0 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa7f8c700 5 monclient: authenticate success, global_id 4730 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa7f8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 1 -- 172.21.15.6:0/1584075381 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9000bb50 con 0x7fdaa0132d00 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa7f8c700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 1 -- 172.21.15.6:0/1584075381 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda90011420 con 0x7fdaa0132d00 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa7f8c700 4 set_mon_vals no callback set 2023-09-30T00:49:19.847 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.837+0000 7fdaa4d26700 10 monclient: dump: 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:19.848 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 10 monclient: shutdown 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/1584075381 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 msgr2=0x7fdaa0137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/1584075381 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 0x7fdaa0137620 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fda90009910 tx=0x7fda900091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/1584075381 shutdown_connections 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/1584075381 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0132d00 0x7fdaa0137620 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/1584075381 >> 172.21.15.6:0/1584075381 conn(0x7fdaa00a5480 msgr2=0x7fdaa00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/1584075381 shutdown_connections 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/1584075381 wait complete. 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 10 monclient: build_initial_monmap 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.838+0000 7fdaa7f8c700 10 monclient: monmap: 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:19.849 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:19.850 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 Processor -- start 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 -- start start 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient: init 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdaa0098670 0x7fdaa009efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): picked mon.noname-b con 0x7fdaa0098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 0x7fdaa0133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.851 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): picked mon.noname-a con 0x7fdaa0132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 0x7fdaa01342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): picked mon.noname-c con 0x7fdaa0137b60 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa012ed90 con 0x7fdaa0132d00 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00a1500 con 0x7fdaa0098670 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:19.852 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaa00a33b0 con 0x7fdaa0137b60 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa7f8c700 10 monclient(hunting): authenticate will time out at 1556.137827s 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa6529700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 0x7fdaa01342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa5527700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 0x7fdaa0133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa6529700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 0x7fdaa01342f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:60504/0 (socket says 172.21.15.6:60504) 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa6529700 1 -- 172.21.15.6:0/3954112507 learned_addr learned my addr 172.21.15.6:0/3954112507 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:19.853 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): get_auth_request con 0x7fdaa0137b60 auth_method 0 2023-09-30T00:49:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.839+0000 7fdaa5527700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 0x7fdaa0133db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50866/0 (socket says 172.21.15.6:50866) 2023-09-30T00:49:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): get_auth_request con 0x7fdaa0132d00 auth_method 0 2023-09-30T00:49:19.854 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdaa0098670 0x7fdaa009efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): get_auth_request con 0x7fdaa0098670 auth_method 0 2023-09-30T00:49:19.855 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.856 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5d28700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient(hunting): handle_auth_done global_id 4733 payload 824 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 1 -- 172.21.15.6:0/3954112507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdaa0098670 msgr2=0x7fdaa009efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:19.857 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdaa0098670 0x7fdaa009efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 1 -- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 msgr2=0x7fdaa0133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 0x7fdaa0133db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient: _finish_hunting 0 2023-09-30T00:49:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 1 monclient: found mon.noname-c 2023-09-30T00:49:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:19.858 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa6529700 1 -- 172.21.15.6:0/3954112507 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90009420 con 0x7fdaa0137b60 2023-09-30T00:49:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.840+0000 7fdaa5527700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 0x7fdaa0133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fdaa6529700 1 -- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 msgr2=0x7fdaa01342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fdaa6529700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 0x7fdaa01342f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda9800c860 tx=0x7fda98002680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98012840 con 0x7fdaa0137b60 2023-09-30T00:49:19.859 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: dump: 2023-09-30T00:49:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.860 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fdaa7f8c700 5 monclient: authenticate success, global_id 4733 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda980129a0 con 0x7fdaa0137b60 2023-09-30T00:49:19.861 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fdaa7f8c700 10 monclient: _renew_subs 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fdaa7f8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaa01336b0 con 0x7fdaa0137b60 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda98012d60 con 0x7fdaa0137b60 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda95ffb700 10 monclient: dump: 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.862 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.843+0000 7fdaa7f8c700 10 monclient: _renew_subs 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.842+0000 7fda97fff700 4 set_mon_vals no callback set 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.843+0000 7fdaa7f8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.843+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaa01370c0 con 0x7fdaa0137b60 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.843+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fda98011590 con 0x7fdaa0137b60 2023-09-30T00:49:19.863 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fda95ffb700 1 --2- 172.21.15.6:0/3954112507 >> v2:172.21.15.22:6812/142603 conn(0x7fda840357a0 0x7fda84037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fdaa7f8c700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fda98047c90 con 0x7fdaa0137b60 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fdaa7f8c700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fdaa7f8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaa0072b00 con 0x7fdaa0137b60 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fdaa5d28700 1 --2- 172.21.15.6:0/3954112507 >> v2:172.21.15.22:6812/142603 conn(0x7fda840357a0 0x7fda84037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:19.864 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.844+0000 7fdaa5d28700 10 monclient: get_auth_request con 0x7fda840357a0 auth_method 0 2023-09-30T00:49:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.847+0000 7fdaa5d28700 1 --2- 172.21.15.6:0/3954112507 >> v2:172.21.15.22:6812/142603 conn(0x7fda840357a0 0x7fda84037c50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fda9c007880 tx=0x7fda9c007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.849+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fda98016070 con 0x7fdaa0137b60 2023-09-30T00:49:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.849+0000 7fda95ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:19.865 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:19.849+0000 7fda95ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:20.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fdaa7f8c700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:20.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fdaa7f8c700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:20.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fdaa7f8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:20.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdaa012fc30 con 0x7fdaa0137b60 2023-09-30T00:49:20.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fda95ffb700 1 -- 172.21.15.6:0/3954112507 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fda980106a0 con 0x7fdaa0137b60 2023-09-30T00:49:20.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fda95ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:20.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.092+0000 7fda95ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:20.094 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:20.094 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:20.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 >> v2:172.21.15.22:6812/142603 conn(0x7fda840357a0 msgr2=0x7fda84037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:20.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/3954112507 >> v2:172.21.15.22:6812/142603 conn(0x7fda840357a0 0x7fda84037c50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fda9c007880 tx=0x7fda9c007f90 comp rx=0 tx=0).stop 2023-09-30T00:49:20.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 10 monclient: shutdown 2023-09-30T00:49:20.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 msgr2=0x7fdaa01342f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 0x7fdaa01342f0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fda9800c860 tx=0x7fda98002680 comp rx=0 tx=0).stop 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 shutdown_connections 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/3954112507 >> v2:172.21.15.22:6812/142603 conn(0x7fda840357a0 0x7fda84037c50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fdaa0137b60 0x7fdaa01342f0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fdaa0098670 0x7fdaa009efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 --2- 172.21.15.6:0/3954112507 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fdaa0132d00 0x7fdaa0133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.094+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 >> 172.21.15.6:0/3954112507 conn(0x7fdaa00a5480 msgr2=0x7fdaa0131af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.095+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 shutdown_connections 2023-09-30T00:49:20.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:20.095+0000 7fdaa7f8c700 1 -- 172.21.15.6:0/3954112507 wait complete. 2023-09-30T00:49:23.109 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.247+0000 7f076201e700 10 monclient: get_monmap_and_config 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.247+0000 7f076201e700 10 monclient: build_initial_monmap 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.247+0000 7f076201e700 10 monclient: monmap: 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:23.248317+0000 2023-09-30T00:49:23.248 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:23.248317+0000 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:23.249 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.247+0000 7f076201e700 1 Processor -- start 2023-09-30T00:49:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.247+0000 7f076201e700 1 -- start start 2023-09-30T00:49:23.250 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.247+0000 7f076201e700 10 monclient: init 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f075c098670 0x7f075c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): picked mon.noname-a con 0x7f075c098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f075c098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 0x7f075c12f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): picked mon.noname-b con 0x7f075c098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:23.254 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f075c0a1000 con 0x7f075c098a40 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f075c0a0a50 con 0x7f075c098670 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0a0730 con 0x7f075c098e90 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f076201e700 10 monclient(hunting): authenticate will time out at 1559.546451s 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f075c098670 0x7f075c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 0x7f075c12f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f075c098670 0x7f075c132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50878/0 (socket says 172.21.15.6:50878) 2023-09-30T00:49:23.255 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 0x7f075c12f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:60508/0 (socket says 172.21.15.6:60508) 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 1 -- 172.21.15.6:0/3209572378 learned_addr learned my addr 172.21.15.6:0/3209572378 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 10 monclient(hunting): get_auth_request con 0x7f075c098670 auth_method 0 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075affd700 1 --1- 172.21.15.6:0/3209572378 >> v1:172.21.15.6:6789/0 conn(0x7f075c098a40 0x7f075c12ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:41584/0 (socket says 172.21.15.6:41584) 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): get_auth_request con 0x7f075c098e90 auth_method 0 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.256 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient(hunting): handle_auth_done global_id 4736 payload 274 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 -- 172.21.15.6:0/3209572378 >> v2:172.21.15.6:3300/0 conn(0x7f075c098670 msgr2=0x7f075c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 --2- 172.21.15.6:0/3209572378 >> v2:172.21.15.6:3300/0 conn(0x7f075c098670 0x7f075c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 -- 172.21.15.6:0/3209572378 >> v1:172.21.15.6:6789/0 conn(0x7f075c098a40 legacy=0x7f075c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 monclient: found mon.noname-b 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:23.257 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.248+0000 7f075bfff700 1 -- 172.21.15.6:0/3209572378 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c068df0 con 0x7f075c098e90 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075bfff700 1 -- 172.21.15.6:0/3209572378 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 msgr2=0x7f075c12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075bfff700 1 --2- 172.21.15.6:0/3209572378 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 0x7f075c12f4f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0754009e60 tx=0x7f0754009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87f73254cc3820ef server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 1 -- 172.21.15.6:0/3209572378 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f075400e070 con 0x7f075c098e90 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: dump: 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.258 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 1 -- 172.21.15.6:0/3209572378 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f075400be00 con 0x7f075c098e90 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 5 monclient: authenticate success, global_id 4736 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:23.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 1 -- 172.21.15.6:0/3209572378 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0754012660 con 0x7f075c098e90 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 4 set_mon_vals no callback set 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f075a7fc700 10 monclient: dump: 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:23.260 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 10 monclient: shutdown 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 1 -- 172.21.15.6:0/3209572378 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 msgr2=0x7f075c12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 1 --2- 172.21.15.6:0/3209572378 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 0x7f075c12f4f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f0754009e60 tx=0x7f0754009780 comp rx=0 tx=0).stop 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.251+0000 7f076201e700 1 -- 172.21.15.6:0/3209572378 shutdown_connections 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 1 --2- 172.21.15.6:0/3209572378 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098e90 0x7f075c12f4f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 1 --2- 172.21.15.6:0/3209572378 >> v2:172.21.15.6:3300/0 conn(0x7f075c098670 0x7f075c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 1 -- 172.21.15.6:0/3209572378 >> 172.21.15.6:0/3209572378 conn(0x7f075c0a5480 msgr2=0x7f075c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:23.261 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 1 -- 172.21.15.6:0/3209572378 shutdown_connections 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 1 -- 172.21.15.6:0/3209572378 wait complete. 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 10 monclient: build_initial_monmap 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.252+0000 7f076201e700 10 monclient: monmap: 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:23.262 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:23.263 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.253+0000 7f076201e700 1 Processor -- start 2023-09-30T00:49:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.253+0000 7f076201e700 1 -- start start 2023-09-30T00:49:23.263 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.253+0000 7f076201e700 10 monclient: init 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 0x7f075c141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): picked mon.noname-a con 0x7f075c098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 0x7f075c09ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): picked mon.noname-c con 0x7f075c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f075c098e90 0x7f075c0994d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): picked mon.noname-b con 0x7f075c098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:23.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c09d1c0 con 0x7f075c098670 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0a1500 con 0x7f075c098e90 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f075c0a3a80 con 0x7f075c098a40 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f076201e700 10 monclient(hunting): authenticate will time out at 1559.552359s 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 0x7f075c141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 0x7f075c141840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:50894/0 (socket says 172.21.15.6:50894) 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 1 -- 172.21.15.6:0/3668459736 learned_addr learned my addr 172.21.15.6:0/3668459736 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:23.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): get_auth_request con 0x7f075c098670 auth_method 0 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 0x7f075c09ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): get_auth_request con 0x7f075c098a40 auth_method 0 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient(hunting): handle_auth_done global_id 4739 payload 824 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075bfff700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f075c098e90 0x7f075c0994d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 -- 172.21.15.6:0/3668459736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f075c098e90 msgr2=0x7f075c0994d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f075c098e90 0x7f075c0994d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 -- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 msgr2=0x7f075c141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 0x7f075c141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 monclient: found mon.noname-c 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:23.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075affd700 1 -- 172.21.15.6:0/3668459736 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0754009a20 con 0x7f075c098a40 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075bfff700 10 monclient: get_auth_request con 0x7f075c098e90 auth_method 0 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075b7fe700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 0x7f075c141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.254+0000 7f075bfff700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f075c098e90 0x7f075c0994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f075affd700 1 -- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 msgr2=0x7f075c09ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f075affd700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 0x7f075c09ebb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f074400c590 tx=0x7f0744007ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744005c80 con 0x7f075c098a40 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: dump: 2023-09-30T00:49:23.271 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:23.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0744005e00 con 0x7f075c098a40 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 5 monclient: authenticate success, global_id 4739 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 10 monclient: _renew_subs 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f075c09af60 con 0x7f075c098a40 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0744007320 con 0x7f075c098a40 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f07537fe700 10 monclient: dump: 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:23.273 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f0759ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 10 monclient: _renew_subs 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:23.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.257+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f075c09bb10 con 0x7f075c098a40 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.258+0000 7f076201e700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.258+0000 7f076201e700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.258+0000 7f076201e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.258+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f075c072b00 con 0x7f075c098a40 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.259+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f074401b3b0 con 0x7f075c098a40 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.259+0000 7f07537fe700 1 --2- 172.21.15.6:0/3668459736 >> v2:172.21.15.22:6812/142603 conn(0x7f073c0357a0 0x7f073c037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.259+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0744047280 con 0x7f075c098a40 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.259+0000 7f075b7fe700 1 --2- 172.21.15.6:0/3668459736 >> v2:172.21.15.22:6812/142603 conn(0x7f073c0357a0 0x7f073c037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.259+0000 7f075b7fe700 10 monclient: get_auth_request con 0x7f073c0357a0 auth_method 0 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.262+0000 7f075b7fe700 1 --2- 172.21.15.6:0/3668459736 >> v2:172.21.15.22:6812/142603 conn(0x7f073c0357a0 0x7f073c037c50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f074c006b00 tx=0x7f074c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.265+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0744018be0 con 0x7f075c098a40 2023-09-30T00:49:23.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.265+0000 7f07537fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:23.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.265+0000 7f07537fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f076201e700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f076201e700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f076201e700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f075c0a2bc0 con 0x7f075c098a40 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f07537fe700 1 -- 172.21.15.6:0/3668459736 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f0744017030 con 0x7f075c098a40 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f07537fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:23.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.510+0000 7f07537fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:23.512 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:23.512 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:23.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 >> v2:172.21.15.22:6812/142603 conn(0x7f073c0357a0 msgr2=0x7f073c037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 --2- 172.21.15.6:0/3668459736 >> v2:172.21.15.22:6812/142603 conn(0x7f073c0357a0 0x7f073c037c50 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f074c006b00 tx=0x7f074c008040 comp rx=0 tx=0).stop 2023-09-30T00:49:23.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 10 monclient: shutdown 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 msgr2=0x7f075c09ebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 0x7f075c09ebb0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f074400c590 tx=0x7f0744007ed0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 shutdown_connections 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 --2- 172.21.15.6:0/3668459736 >> v2:172.21.15.22:6812/142603 conn(0x7f073c0357a0 0x7f073c037c50 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f075c098e90 0x7f075c0994d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.512+0000 7f076201e700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f075c098670 0x7f075c141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.513+0000 7f076201e700 1 --2- 172.21.15.6:0/3668459736 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f075c098a40 0x7f075c09ebb0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.513+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 >> 172.21.15.6:0/3668459736 conn(0x7f075c0a5480 msgr2=0x7f075c069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.513+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 shutdown_connections 2023-09-30T00:49:23.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:23.513+0000 7f076201e700 1 -- 172.21.15.6:0/3668459736 wait complete. 2023-09-30T00:49:26.527 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.666+0000 7fab80ac5700 10 monclient: get_monmap_and_config 2023-09-30T00:49:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.666+0000 7fab80ac5700 10 monclient: build_initial_monmap 2023-09-30T00:49:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.666+0000 7fab80ac5700 10 monclient: monmap: 2023-09-30T00:49:26.666 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:26.666379+0000 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:26.666379+0000 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:26.667 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:26.668 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.666+0000 7fab80ac5700 1 Processor -- start 2023-09-30T00:49:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.666+0000 7fab80ac5700 1 -- start start 2023-09-30T00:49:26.668 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.666+0000 7fab80ac5700 10 monclient: init 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient: _add_conns ranks=[2,1,0,4,3] 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 0x7fab7c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient(hunting): picked mon.noname-b con 0x7fab7c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fab7c098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fab7c098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:26.677 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c0a1000 con 0x7fab7c098e90 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab7c12fc00 con 0x7fab7c098a40 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c0a0a50 con 0x7fab7c098670 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab80ac5700 10 monclient(hunting): authenticate will time out at 1562.964533s 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 0x7fab7c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 0x7fab7c132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:44082/0 (socket says 172.21.15.6:44082) 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 1 -- 172.21.15.6:0/3080674691 learned_addr learned my addr 172.21.15.6:0/3080674691 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): get_auth_request con 0x7fab7c098670 auth_method 0 2023-09-30T00:49:26.678 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7ad9d700 1 --1- 172.21.15.6:0/3080674691 >> v1:172.21.15.6:6789/0 conn(0x7fab7c098e90 0x7fab7c12f4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53560/0 (socket says 172.21.15.6:53560) 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.667+0000 7fab7a59c700 10 monclient(hunting): handle_auth_done global_id 4742 payload 274 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 1 -- 172.21.15.6:0/3080674691 >> v1:172.21.15.22:6789/0 conn(0x7fab7c098a40 legacy=0x7fab7c12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 1 -- 172.21.15.6:0/3080674691 >> v1:172.21.15.6:6789/0 conn(0x7fab7c098e90 legacy=0x7fab7c12f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:26.679 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 10 monclient: _finish_hunting 0 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 1 monclient: found mon.noname-b 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.668+0000 7fab7a59c700 1 -- 172.21.15.6:0/3080674691 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab7c0a0730 con 0x7fab7c098670 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7a59c700 1 -- 172.21.15.6:0/3080674691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 msgr2=0x7fab7c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7a59c700 1 --2- 172.21.15.6:0/3080674691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 0x7fab7c132d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0099e0 tx=0x7fab6c009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d93d67c937e0511 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 1 -- 172.21.15.6:0/3080674691 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c00bad0 con 0x7fab7c098670 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:26.680 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: dump: 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:26.681 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: _finish_auth 0 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab80ac5700 5 monclient: authenticate success, global_id 4742 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 1 -- 172.21.15.6:0/3080674691 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab6c00bc30 con 0x7fab7c098670 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab80ac5700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab80ac5700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab80ac5700 4 set_mon_vals no callback set 2023-09-30T00:49:26.682 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 1 -- 172.21.15.6:0/3080674691 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c011420 con 0x7fab7c098670 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.670+0000 7fab7959a700 10 monclient: dump: 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.683 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 10 monclient: shutdown 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 -- 172.21.15.6:0/3080674691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 msgr2=0x7fab7c132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 --2- 172.21.15.6:0/3080674691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 0x7fab7c132d00 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0099e0 tx=0x7fab6c009250 comp rx=0 tx=0).stop 2023-09-30T00:49:26.684 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 -- 172.21.15.6:0/3080674691 shutdown_connections 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 --2- 172.21.15.6:0/3080674691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098670 0x7fab7c132d00 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 -- 172.21.15.6:0/3080674691 >> 172.21.15.6:0/3080674691 conn(0x7fab7c0a5480 msgr2=0x7fab7c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 -- 172.21.15.6:0/3080674691 shutdown_connections 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 1 -- 172.21.15.6:0/3080674691 wait complete. 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 10 monclient: build_initial_monmap 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.671+0000 7fab80ac5700 10 monclient: monmap: 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:26.685 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 Processor -- start 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 -- start start 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient: init 2023-09-30T00:49:26.686 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab7c098670 0x7fab7c141840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): picked mon.noname-b con 0x7fab7c098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 0x7fab7c12c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): picked mon.noname-c con 0x7fab7c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 0x7fab7c12c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): picked mon.noname-a con 0x7fab7c098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c142740 con 0x7fab7c098e90 2023-09-30T00:49:26.687 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c09e6e0 con 0x7fab7c098670 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab7c12ed70 con 0x7fab7c098a40 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab80ac5700 10 monclient(hunting): authenticate will time out at 1562.969710s 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7ad9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 0x7fab7c12c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 0x7fab7c12c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 0x7fab7c12c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:44098/0 (socket says 172.21.15.6:44098) 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7ad9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 0x7fab7c12c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46536/0 (socket says 172.21.15.6:46536) 2023-09-30T00:49:26.688 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 1 -- 172.21.15.6:0/2489966249 learned_addr learned my addr 172.21.15.6:0/2489966249 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): get_auth_request con 0x7fab7c098a40 auth_method 0 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7ad9d700 10 monclient(hunting): get_auth_request con 0x7fab7c098e90 auth_method 0 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7ad9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.689 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7ad9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7ad9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7a59c700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab7c098670 0x7fab7c141840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7a59c700 10 monclient(hunting): get_auth_request con 0x7fab7c098670 auth_method 0 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7a59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7a59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab7a59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.672+0000 7fab79d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.690 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7ad9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7ad9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7ad9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7a59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7a59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7a59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 10 monclient(hunting): handle_auth_done global_id 4745 payload 824 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 1 -- 172.21.15.6:0/2489966249 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab7c098670 msgr2=0x7fab7c141840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab7c098670 0x7fab7c141840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.691 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 1 -- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 msgr2=0x7fab7c12c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 0x7fab7c12c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 1 monclient: found mon.noname-c 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab79d9b700 1 -- 172.21.15.6:0/2489966249 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab6c0094f0 con 0x7fab7c098a40 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.673+0000 7fab7ad9d700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 0x7fab7c12c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab79d9b700 1 -- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 msgr2=0x7fab7c12c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:26.692 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab79d9b700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 0x7fab7c12c200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab7000c600 tx=0x7fab70007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab700117a0 con 0x7fab7c098a40 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: dump: 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.693 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab70011900 con 0x7fab7c098a40 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:26.694 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab70011cc0 con 0x7fab7c098a40 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 5 monclient: authenticate success, global_id 4745 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 10 monclient: _renew_subs 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab7c11c350 con 0x7fab7c098a40 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab78d99700 4 set_mon_vals no callback set 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab6a7fc700 10 monclient: dump: 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:26.695 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:26.696 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 10 monclient: _renew_subs 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.675+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab7c141d80 con 0x7fab7c098a40 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fab70012630 con 0x7fab7c098a40 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab6a7fc700 1 --2- 172.21.15.6:0/2489966249 >> v2:172.21.15.22:6812/142603 conn(0x7fab5c0357f0 0x7fab5c037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fab700475f0 con 0x7fab7c098a40 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab80ac5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab80ac5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab80ac5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.676+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab7c141fa0 con 0x7fab7c098a40 2023-09-30T00:49:26.697 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.677+0000 7fab7a59c700 1 --2- 172.21.15.6:0/2489966249 >> v2:172.21.15.22:6812/142603 conn(0x7fab5c0357f0 0x7fab5c037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.677+0000 7fab7a59c700 10 monclient: get_auth_request con 0x7fab5c0357f0 auth_method 0 2023-09-30T00:49:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.679+0000 7fab7a59c700 1 --2- 172.21.15.6:0/2489966249 >> v2:172.21.15.22:6812/142603 conn(0x7fab5c0357f0 0x7fab5c037ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00bfd0 tx=0x7fab6c016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.681+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fab7001c460 con 0x7fab7c098a40 2023-09-30T00:49:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.681+0000 7fab6a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:26.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.681+0000 7fab6a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab80ac5700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab80ac5700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:26.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab80ac5700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fab7c072b00 con 0x7fab7c098a40 2023-09-30T00:49:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab6a7fc700 1 -- 172.21.15.6:0/2489966249 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fab700102e0 con 0x7fab7c098a40 2023-09-30T00:49:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab6a7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:26.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.929+0000 7fab6a7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:26.930 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:26.930 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 >> v2:172.21.15.22:6812/142603 conn(0x7fab5c0357f0 msgr2=0x7fab5c037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 --2- 172.21.15.6:0/2489966249 >> v2:172.21.15.22:6812/142603 conn(0x7fab5c0357f0 0x7fab5c037ca0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fab6c00bfd0 tx=0x7fab6c016000 comp rx=0 tx=0).stop 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 10 monclient: shutdown 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 msgr2=0x7fab7c12c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 0x7fab7c12c200 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fab7000c600 tx=0x7fab70007f60 comp rx=0 tx=0).stop 2023-09-30T00:49:26.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 shutdown_connections 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 --2- 172.21.15.6:0/2489966249 >> v2:172.21.15.22:6812/142603 conn(0x7fab5c0357f0 0x7fab5c037ca0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fab7c098e90 0x7fab7c12c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fab7c098670 0x7fab7c141840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 --2- 172.21.15.6:0/2489966249 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fab7c098a40 0x7fab7c12c200 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 >> 172.21.15.6:0/2489966249 conn(0x7fab7c0a5480 msgr2=0x7fab7c12e380 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 shutdown_connections 2023-09-30T00:49:26.933 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:26.932+0000 7fab80ac5700 1 -- 172.21.15.6:0/2489966249 wait complete. 2023-09-30T00:49:29.945 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.084+0000 7f6434456700 10 monclient: get_monmap_and_config 2023-09-30T00:49:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.084+0000 7f6434456700 10 monclient: build_initial_monmap 2023-09-30T00:49:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.084+0000 7f6434456700 10 monclient: monmap: 2023-09-30T00:49:30.084 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:30.084852+0000 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:30.084852+0000 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:30.085 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:30.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.085+0000 7f6434456700 1 Processor -- start 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.085+0000 7f6434456700 1 -- start start 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.085+0000 7f6434456700 10 monclient: init 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient: _add_conns ranks=[1,0,4,2,3] 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f642c098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f642c132d00 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f642c138d40 0x7f642c13b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient(hunting): picked mon.noname-c con 0x7f642c138d40 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0a1000 con 0x7f642c132d00 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f642c0a0730 con 0x7f642c098670 2023-09-30T00:49:30.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f642c099150 con 0x7f642c138d40 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f6434456700 10 monclient(hunting): authenticate will time out at 1566.383335s 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64319f1700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f642c132d00 0x7f642c137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53566/0 (socket says 172.21.15.6:53566) 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64319f1700 1 -- 172.21.15.6:0/1561237137 learned_addr learned my addr 172.21.15.6:0/1561237137 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 1 --2- 172.21.15.6:0/1561237137 >> v2:172.21.15.22:3300/0 conn(0x7f642c138d40 0x7f642c13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): get_auth_request con 0x7f642c138d40 auth_method 0 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient(hunting): handle_auth_done global_id 4726 payload 274 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 1 -- 172.21.15.6:0/1561237137 >> v1:172.21.15.22:6789/0 conn(0x7f642c098670 legacy=0x7f642c098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 1 -- 172.21.15.6:0/1561237137 >> v1:172.21.15.6:6789/0 conn(0x7f642c132d00 legacy=0x7f642c137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient: _finish_hunting 0 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 1 monclient: found mon.noname-c 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.22:3300/0 2023-09-30T00:49:30.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.086+0000 7f64329f3700 1 -- 172.21.15.6:0/1561237137 --> v2:172.21.15.22:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f642c068df0 con 0x7f642c138d40 2023-09-30T00:49:30.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64329f3700 1 -- 172.21.15.6:0/1561237137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c138d40 msgr2=0x7f642c13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:30.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64329f3700 1 --2- 172.21.15.6:0/1561237137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c138d40 0x7f642c13b120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6424009e60 tx=0x7f6424009780 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0489fb59ebaccfa server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:30.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 1 -- 172.21.15.6:0/1561237137 <== mon.1 v2:172.21.15.22:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642400e070 con 0x7f642c138d40 2023-09-30T00:49:30.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: dump: 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:30.094 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: _finish_auth 0 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 1 -- 172.21.15.6:0/1561237137 <== mon.1 v2:172.21.15.22:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424012430 con 0x7f642c138d40 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 5 monclient: authenticate success, global_id 4726 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 4 set_mon_vals no callback set 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 1 -- 172.21.15.6:0/1561237137 <== mon.1 v2:172.21.15.22:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240127d0 con 0x7f642c138d40 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.095 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: got monmap 1 from mon.b (according to old e1) 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f64311f0700 10 monclient: dump: 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 10 monclient: shutdown 2023-09-30T00:49:30.096 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 1 -- 172.21.15.6:0/1561237137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c138d40 msgr2=0x7f642c13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.089+0000 7f6434456700 1 --2- 172.21.15.6:0/1561237137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c138d40 0x7f642c13b120 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f6424009e60 tx=0x7f6424009780 comp rx=0 tx=0).stop 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 1 -- 172.21.15.6:0/1561237137 shutdown_connections 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 1 --2- 172.21.15.6:0/1561237137 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c138d40 0x7f642c13b120 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 1 -- 172.21.15.6:0/1561237137 >> 172.21.15.6:0/1561237137 conn(0x7f642c0a5480 msgr2=0x7f642c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 1 -- 172.21.15.6:0/1561237137 shutdown_connections 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 1 -- 172.21.15.6:0/1561237137 wait complete. 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 10 monclient: build_initial_monmap 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 10 monclient: monmap: 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:30.097 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.090+0000 7f6434456700 1 Processor -- start 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 -- start start 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient: init 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:30.098 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f642c098670 0x7f642c1416e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): picked mon.noname-c con 0x7f642c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c132d00 0x7f642c1383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): picked mon.noname-b con 0x7f642c132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 0x7f642c13e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): picked mon.noname-a con 0x7f642c138d40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1335f0 con 0x7f642c138d40 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f642c1338d0 con 0x7f642c132d00 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f642c133bb0 con 0x7f642c098670 2023-09-30T00:49:30.099 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f6434456700 10 monclient(hunting): authenticate will time out at 1566.388503s 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 0x7f642c13e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 0x7f642c13e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46544/0 (socket says 172.21.15.6:46544) 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 -- 172.21.15.6:0/3562504146 learned_addr learned my addr 172.21.15.6:0/3562504146 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f642c098670 0x7f642c1416e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): get_auth_request con 0x7f642c138d40 auth_method 0 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): get_auth_request con 0x7f642c098670 auth_method 0 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.100 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64321f2700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64319f1700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c132d00 0x7f642c1383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64319f1700 10 monclient(hunting): get_auth_request con 0x7f642c132d00 auth_method 0 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64319f1700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64319f1700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:30.101 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64319f1700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient(hunting): handle_auth_done global_id 4713 payload 824 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 -- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f642c098670 msgr2=0x7f642c1416e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f642c098670 0x7f642c1416e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 -- 172.21.15.6:0/3562504146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c132d00 msgr2=0x7f642c1383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c132d00 0x7f642c1383d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient: _finish_hunting 0 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 monclient: found mon.noname-a 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.091+0000 7f64329f3700 1 -- 172.21.15.6:0/3562504146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6424009a20 con 0x7f642c138d40 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.092+0000 7f64319f1700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c132d00 0x7f642c1383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:30.102 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.092+0000 7f64321f2700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f642c098670 0x7f642c1416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:30.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64329f3700 1 -- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 msgr2=0x7f642c13e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:30.103 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64329f3700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 0x7f642c13e1d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f642c06a6b0 tx=0x7f6424018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424026430 con 0x7f642c138d40 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: dump: 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.105 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6424021070 con 0x7f642c138d40 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 5 monclient: authenticate success, global_id 4713 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 10 monclient: _renew_subs 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f642c12f1f0 con 0x7f642c138d40 2023-09-30T00:49:30.106 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64240267e0 con 0x7f642c138d40 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64227fc700 10 monclient: dump: 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:30.107 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f64309ef700 4 set_mon_vals no callback set 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 10 monclient: _renew_subs 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.094+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f642c12e050 con 0x7f642c138d40 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6424023030 con 0x7f642c138d40 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f6434456700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f6434456700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f6434456700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f642c133730 con 0x7f642c138d40 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f64227fc700 1 --2- 172.21.15.6:0/3562504146 >> v2:172.21.15.22:6812/142603 conn(0x7f64100357a0 0x7f6410037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:30.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.095+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6424049c90 con 0x7f642c138d40 2023-09-30T00:49:30.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.096+0000 7f64321f2700 1 --2- 172.21.15.6:0/3562504146 >> v2:172.21.15.22:6812/142603 conn(0x7f64100357a0 0x7f6410037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:30.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.096+0000 7f64321f2700 10 monclient: get_auth_request con 0x7f64100357a0 auth_method 0 2023-09-30T00:49:30.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.098+0000 7f64321f2700 1 --2- 172.21.15.6:0/3562504146 >> v2:172.21.15.22:6812/142603 conn(0x7f64100357a0 0x7f6410037c50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6428002e00 tx=0x7f6428007ea0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:30.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.100+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f642c133730 con 0x7f642c138d40 2023-09-30T00:49:30.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.100+0000 7f64227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:30.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.100+0000 7f64227fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.350+0000 7f6434456700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:30.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.350+0000 7f6434456700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.350+0000 7f6434456700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.350+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f642c133730 con 0x7f642c138d40 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.351+0000 7f64227fc700 1 -- 172.21.15.6:0/3562504146 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f642c133730 con 0x7f642c138d40 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.351+0000 7f64227fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.351+0000 7f64227fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:30.351 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:30.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 >> v2:172.21.15.22:6812/142603 conn(0x7f64100357a0 msgr2=0x7f6410037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 --2- 172.21.15.6:0/3562504146 >> v2:172.21.15.22:6812/142603 conn(0x7f64100357a0 0x7f6410037c50 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f6428002e00 tx=0x7f6428007ea0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 10 monclient: shutdown 2023-09-30T00:49:30.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 msgr2=0x7f642c13e1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 0x7f642c13e1d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f642c06a6b0 tx=0x7f6424018000 comp rx=0 tx=0).stop 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 shutdown_connections 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 --2- 172.21.15.6:0/3562504146 >> v2:172.21.15.22:6812/142603 conn(0x7f64100357a0 0x7f6410037c50 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f642c138d40 0x7f642c13e1d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f642c098670 0x7f642c1416e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 --2- 172.21.15.6:0/3562504146 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f642c132d00 0x7f642c1383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 >> 172.21.15.6:0/3562504146 conn(0x7f642c0a5480 msgr2=0x7f642c0a1ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 shutdown_connections 2023-09-30T00:49:30.354 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:30.353+0000 7f6434456700 1 -- 172.21.15.6:0/3562504146 wait complete. 2023-09-30T00:49:33.367 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.505+0000 7f7238fc2700 10 monclient: get_monmap_and_config 2023-09-30T00:49:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.505+0000 7f7238fc2700 10 monclient: build_initial_monmap 2023-09-30T00:49:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.505+0000 7f7238fc2700 10 monclient: monmap: 2023-09-30T00:49:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:33.505 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:33.505999+0000 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:33.505999+0000 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:33.506 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:33.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.506+0000 7f7238fc2700 1 Processor -- start 2023-09-30T00:49:33.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.506+0000 7f7238fc2700 1 -- start start 2023-09-30T00:49:33.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.506+0000 7f7238fc2700 10 monclient: init 2023-09-30T00:49:33.510 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient: _add_conns ranks=[0,2,3,4,1] 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f7234098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 0x7f7234137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): picked mon.noname-b con 0x7f7234132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7234137b60 0x7f7234139f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): picked mon.noname-a con 0x7f7234137b60 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72340a1000 con 0x7f7234098670 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7234099150 con 0x7f7234137b60 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:33.511 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72340a0a50 con 0x7f7234132d00 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7238fc2700 10 monclient(hunting): authenticate will time out at 1569.804220s 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 0x7f7234137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7232d9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7234137b60 0x7f7234139f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 0x7f7234137620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:44126/0 (socket says 172.21.15.6:44126) 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7232d9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f7234137b60 0x7f7234139f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:46550/0 (socket says 172.21.15.6:46550) 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 -- 172.21.15.6:0/2281756727 learned_addr learned my addr 172.21.15.6:0/2281756727 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): get_auth_request con 0x7f7234132d00 auth_method 0 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f723259c700 1 --1- 172.21.15.6:0/2281756727 >> v1:172.21.15.6:6789/0 conn(0x7f7234098670 0x7f7234098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:53574/0 (socket says 172.21.15.6:53574) 2023-09-30T00:49:33.512 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7232d9d700 10 monclient(hunting): get_auth_request con 0x7f7234137b60 auth_method 0 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7232d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7232d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7232d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient(hunting): handle_auth_done global_id 4751 payload 274 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 -- 172.21.15.6:0/2281756727 >> v2:172.21.15.6:3300/0 conn(0x7f7234137b60 msgr2=0x7f7234139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 --2- 172.21.15.6:0/2281756727 >> v2:172.21.15.6:3300/0 conn(0x7f7234137b60 0x7f7234139f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.513 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 -- 172.21.15.6:0/2281756727 >> v1:172.21.15.6:6789/0 conn(0x7f7234098670 legacy=0x7f7234098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 monclient: found mon.noname-b 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.507+0000 7f7231d9b700 1 -- 172.21.15.6:0/2281756727 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72340a0730 con 0x7f7234132d00 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.509+0000 7f7231d9b700 1 -- 172.21.15.6:0/2281756727 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 msgr2=0x7f7234137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.509+0000 7f7231d9b700 1 --2- 172.21.15.6:0/2281756727 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 0x7f7234137620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f72280099b0 tx=0x7f7228009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71fe5e5b1dd1947a server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 1 -- 172.21.15.6:0/2281756727 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722800ba30 con 0x7f7234132d00 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:33.514 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: dump: 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: _finish_auth 0 2023-09-30T00:49:33.515 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 1 -- 172.21.15.6:0/2281756727 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722800bb90 con 0x7f7234132d00 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 5 monclient: authenticate success, global_id 4751 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 1 -- 172.21.15.6:0/2281756727 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7228011420 con 0x7f7234132d00 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 4 set_mon_vals no callback set 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f723159a700 10 monclient: dump: 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:33.516 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 10 monclient: shutdown 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 -- 172.21.15.6:0/2281756727 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 msgr2=0x7f7234137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:33.517 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 --2- 172.21.15.6:0/2281756727 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 0x7f7234137620 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f72280099b0 tx=0x7f7228009250 comp rx=0 tx=0).stop 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 -- 172.21.15.6:0/2281756727 shutdown_connections 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 --2- 172.21.15.6:0/2281756727 >> v2:172.21.15.6:3300/0 conn(0x7f7234137b60 0x7f7234139f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 --2- 172.21.15.6:0/2281756727 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234132d00 0x7f7234137620 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 -- 172.21.15.6:0/2281756727 >> 172.21.15.6:0/2281756727 conn(0x7f72340a5480 msgr2=0x7f72340a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 -- 172.21.15.6:0/2281756727 shutdown_connections 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.510+0000 7f7238fc2700 1 -- 172.21.15.6:0/2281756727 wait complete. 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.511+0000 7f7238fc2700 10 monclient: build_initial_monmap 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.511+0000 7f7238fc2700 10 monclient: monmap: 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:33.518 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:33.519 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:33.523 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.511+0000 7f7238fc2700 1 Processor -- start 2023-09-30T00:49:33.523 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.511+0000 7f7238fc2700 1 -- start start 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.511+0000 7f7238fc2700 10 monclient: init 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 0x7f723409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): picked mon.noname-c con 0x7f7234098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7234132d00 0x7f723409cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): picked mon.noname-b con 0x7f7234132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 0x7f723409d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): picked mon.noname-a con 0x7f7234137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:33.524 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72340a1500 con 0x7f7234137b60 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7234137880 con 0x7f7234132d00 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7234133410 con 0x7f7234098670 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7238fc2700 10 monclient(hunting): authenticate will time out at 1569.809262s 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 0x7f723409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 0x7f723409d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 0x7f723409c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:44134/0 (socket says 172.21.15.6:44134) 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 1 -- 172.21.15.6:0/637899199 learned_addr learned my addr 172.21.15.6:0/637899199 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): get_auth_request con 0x7f7234098670 auth_method 0 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:33.525 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): get_auth_request con 0x7f7234137b60 auth_method 0 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:33.526 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7234132d00 0x7f723409cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): get_auth_request con 0x7f7234132d00 auth_method 0 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7231d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient(hunting): handle_auth_done global_id 4719 payload 824 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 -- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 msgr2=0x7f723409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 0x7f723409c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 -- 172.21.15.6:0/637899199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7234132d00 msgr2=0x7f723409cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:33.527 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7234132d00 0x7f723409cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient: _finish_hunting 0 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 monclient: found mon.noname-a 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f7232d9d700 1 -- 172.21.15.6:0/637899199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72280094f0 con 0x7f7234137b60 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.512+0000 7f723259c700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 0x7f723409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.514+0000 7f7232d9d700 1 -- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 msgr2=0x7f723409d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7232d9d700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 0x7f723409d220 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f72340a3350 tx=0x7f722c009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c018070 con 0x7f7234137b60 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:33.528 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: dump: 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:33.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f722c00ecd0 con 0x7f7234137b60 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 5 monclient: authenticate success, global_id 4719 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c014650 con 0x7f7234137b60 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 10 monclient: _renew_subs 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72341336b0 con 0x7f7234137b60 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7230d99700 4 set_mon_vals no callback set 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f72227fc700 10 monclient: dump: 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:33.530 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 10 monclient: _renew_subs 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.515+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723409dc40 con 0x7f7234137b60 2023-09-30T00:49:33.531 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f722c003020 con 0x7f7234137b60 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f7238fc2700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f7238fc2700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f7238fc2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723409f7c0 con 0x7f7234137b60 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f72227fc700 1 --2- 172.21.15.6:0/637899199 >> v2:172.21.15.22:6812/142603 conn(0x7f72140357a0 0x7f7214037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f722c047b10 con 0x7f7234137b60 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f723259c700 1 --2- 172.21.15.6:0/637899199 >> v2:172.21.15.22:6812/142603 conn(0x7f72140357a0 0x7f7214037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.516+0000 7f723259c700 10 monclient: get_auth_request con 0x7f72140357a0 auth_method 0 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.519+0000 7f723259c700 1 --2- 172.21.15.6:0/637899199 >> v2:172.21.15.22:6812/142603 conn(0x7f72140357a0 0x7f7214037c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7224005140 tx=0x7f7224007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.521+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f722c00ee40 con 0x7f7234137b60 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.521+0000 7f72227fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:33.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.521+0000 7f72227fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:33.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f7238fc2700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f7238fc2700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f7238fc2700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7234072b00 con 0x7f7234137b60 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f72227fc700 1 -- 172.21.15.6:0/637899199 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7f722c0122e0 con 0x7f7234137b60 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f72227fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.765+0000 7f72227fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:33.765 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 >> v2:172.21.15.22:6812/142603 conn(0x7f72140357a0 msgr2=0x7f7214037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 --2- 172.21.15.6:0/637899199 >> v2:172.21.15.22:6812/142603 conn(0x7f72140357a0 0x7f7214037c50 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f7224005140 tx=0x7f7224007f90 comp rx=0 tx=0).stop 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 10 monclient: shutdown 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 msgr2=0x7f723409d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 0x7f723409d220 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f72340a3350 tx=0x7f722c009850 comp rx=0 tx=0).stop 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 shutdown_connections 2023-09-30T00:49:33.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 --2- 172.21.15.6:0/637899199 >> v2:172.21.15.22:6812/142603 conn(0x7f72140357a0 0x7f7214037c50 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f7234137b60 0x7f723409d220 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f7234098670 0x7f723409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 --2- 172.21.15.6:0/637899199 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f7234132d00 0x7f723409cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 >> 172.21.15.6:0/637899199 conn(0x7f72340a5480 msgr2=0x7f7234068ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 shutdown_connections 2023-09-30T00:49:33.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:33.767+0000 7f7238fc2700 1 -- 172.21.15.6:0/637899199 wait complete. 2023-09-30T00:49:36.781 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2023-09-30T00:49:36.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.921+0000 7fbd848f6700 10 monclient: get_monmap_and_config 2023-09-30T00:49:36.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.921+0000 7fbd848f6700 10 monclient: build_initial_monmap 2023-09-30T00:49:36.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.921+0000 7fbd848f6700 10 monclient: monmap: 2023-09-30T00:49:36.921 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:36.921953+0000 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:36.921953+0000 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:36.922 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:36.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.922+0000 7fbd848f6700 1 Processor -- start 2023-09-30T00:49:36.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.922+0000 7fbd848f6700 1 -- start start 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.922+0000 7fbd848f6700 10 monclient: init 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient: _add_conns ranks=[4,2,0,3,1] 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7fbd80098670 0x7fbd80132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): picked mon.noname-c con 0x7fbd80098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): picked mon.noname-b con 0x7fbd80098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fbd80098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:36.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbd800a1000 con 0x7fbd80098e90 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbd800a0a50 con 0x7fbd80098670 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd800a0730 con 0x7fbd80098a40 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd848f6700 10 monclient(hunting): authenticate will time out at 1573.220260s 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7ed9d700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fbd80098e90 0x7fbd8012f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33426/0 (socket says 172.21.15.6:33426) 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7ed9d700 1 -- 172.21.15.6:0/412770260 learned_addr learned my addr 172.21.15.6:0/412770260 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): get_auth_request con 0x7fbd80098a40 auth_method 0 2023-09-30T00:49:36.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 1 --2- 172.21.15.6:0/412770260 >> v2:172.21.15.22:3300/0 conn(0x7fbd80098670 0x7fbd80132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): get_auth_request con 0x7fbd80098670 auth_method 0 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:36.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_done global_id 4757 payload 274 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/412770260 >> v2:172.21.15.22:3300/0 conn(0x7fbd80098670 msgr2=0x7fbd80132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 1 --2- 172.21.15.6:0/412770260 >> v2:172.21.15.22:3300/0 conn(0x7fbd80098670 0x7fbd80132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/412770260 >> v1:172.21.15.6:6789/0 conn(0x7fbd80098e90 legacy=0x7fbd8012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 1 monclient: found mon.noname-b 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.923+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/412770260 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80068df0 con 0x7fbd80098a40 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/412770260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 msgr2=0x7fbd8012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:36.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7dd9b700 1 --2- 172.21.15.6:0/412770260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd740098e0 tx=0x7fbd740091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0e88359de706ccb server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 1 -- 172.21.15.6:0/412770260 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74011840 con 0x7fbd80098a40 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: dump: 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:36.930 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: _finish_auth 0 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 1 -- 172.21.15.6:0/412770260 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd740119a0 con 0x7fbd80098a40 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 5 monclient: authenticate success, global_id 4757 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 1 -- 172.21.15.6:0/412770260 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74011d40 con 0x7fbd80098a40 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:36.931 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 4 set_mon_vals no callback set 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd7d59a700 10 monclient: dump: 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:36.932 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:36.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 10 monclient: shutdown 2023-09-30T00:49:36.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:36.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 -- 172.21.15.6:0/412770260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 msgr2=0x7fbd8012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:36.940 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 --2- 172.21.15.6:0/412770260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012ede0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbd740098e0 tx=0x7fbd740091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 -- 172.21.15.6:0/412770260 shutdown_connections 2023-09-30T00:49:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 --2- 172.21.15.6:0/412770260 >> v2:172.21.15.22:3300/0 conn(0x7fbd80098670 0x7fbd80132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 --2- 172.21.15.6:0/412770260 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012ede0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 -- 172.21.15.6:0/412770260 >> 172.21.15.6:0/412770260 conn(0x7fbd800a5480 msgr2=0x7fbd800a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:36.941 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.926+0000 7fbd848f6700 1 -- 172.21.15.6:0/412770260 shutdown_connections 2023-09-30T00:49:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.927+0000 7fbd848f6700 1 -- 172.21.15.6:0/412770260 wait complete. 2023-09-30T00:49:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.927+0000 7fbd848f6700 10 monclient: build_initial_monmap 2023-09-30T00:49:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.927+0000 7fbd848f6700 10 monclient: monmap: 2023-09-30T00:49:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:36.942 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:36.943 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:36.943 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:36.943 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:36.943 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:36.943 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:36.943 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:36.944 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.927+0000 7fbd848f6700 1 Processor -- start 2023-09-30T00:49:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.927+0000 7fbd848f6700 1 -- start start 2023-09-30T00:49:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.927+0000 7fbd848f6700 10 monclient: init 2023-09-30T00:49:36.944 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 0x7fbd80141a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): picked mon.noname-a con 0x7fbd80098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): picked mon.noname-c con 0x7fbd80098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:36.945 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd80098e90 0x7fbd8013e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:36.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): picked mon.noname-b con 0x7fbd80098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:36.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:36.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd80131390 con 0x7fbd80098670 2023-09-30T00:49:36.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:36.946 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8011c4a0 con 0x7fbd80098e90 2023-09-30T00:49:36.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:36.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbd8011c780 con 0x7fbd80098a40 2023-09-30T00:49:36.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:36.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd848f6700 10 monclient(hunting): authenticate will time out at 1573.225481s 2023-09-30T00:49:36.947 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 0x7fbd80141a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 0x7fbd80141a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35670/0 (socket says 172.21.15.6:35670) 2023-09-30T00:49:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 1 -- 172.21.15.6:0/70892033 learned_addr learned my addr 172.21.15.6:0/70892033 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): get_auth_request con 0x7fbd80098670 auth_method 0 2023-09-30T00:49:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:36.948 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:36.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:36.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): get_auth_request con 0x7fbd80098a40 auth_method 0 2023-09-30T00:49:36.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:36.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:36.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:36.949 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:36.950 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7ed9d700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd80098e90 0x7fbd8013e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7ed9d700 10 monclient(hunting): get_auth_request con 0x7fbd80098e90 auth_method 0 2023-09-30T00:49:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:36.951 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient(hunting): handle_auth_done global_id 4760 payload 824 2023-09-30T00:49:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/70892033 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd80098e90 msgr2=0x7fbd8013e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd80098e90 0x7fbd8013e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 msgr2=0x7fbd80141a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 0x7fbd80141a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:36.952 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:49:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 monclient: found mon.noname-c 2023-09-30T00:49:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/70892033 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd74009420 con 0x7fbd80098a40 2023-09-30T00:49:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7e59c700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 0x7fbd80141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:36.953 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.928+0000 7fbd7ed9d700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd80098e90 0x7fbd8013e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd7dd9b700 1 -- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 msgr2=0x7fbd8012c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd7dd9b700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012c350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbd800a1df0 tx=0x7fbd74016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd74000c40 con 0x7fbd80098a40 2023-09-30T00:49:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:36.954 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: dump: 2023-09-30T00:49:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.955 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:36.956 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:36.956 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbd74020070 con 0x7fbd80098a40 2023-09-30T00:49:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:36.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd7401b450 con 0x7fbd80098a40 2023-09-30T00:49:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd6e7fc700 10 monclient: dump: 2023-09-30T00:49:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:36.958 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:36.959 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:36.960 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 5 monclient: authenticate success, global_id 4760 2023-09-30T00:49:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 10 monclient: _renew_subs 2023-09-30T00:49:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:36.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd7cd99700 4 set_mon_vals no callback set 2023-09-30T00:49:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd8011ca60 con 0x7fbd80098a40 2023-09-30T00:49:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 10 monclient: _renew_subs 2023-09-30T00:49:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.931+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd8009f440 con 0x7fbd80098a40 2023-09-30T00:49:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd848f6700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:36.961 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd848f6700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd848f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fbd74022030 con 0x7fbd80098a40 2023-09-30T00:49:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80098170 con 0x7fbd80098a40 2023-09-30T00:49:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd6e7fc700 1 --2- 172.21.15.6:0/70892033 >> v2:172.21.15.22:6812/142603 conn(0x7fbd600357a0 0x7fbd60037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:36.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fbd74049510 con 0x7fbd80098a40 2023-09-30T00:49:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd7e59c700 1 --2- 172.21.15.6:0/70892033 >> v2:172.21.15.22:6812/142603 conn(0x7fbd600357a0 0x7fbd60037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.932+0000 7fbd7e59c700 10 monclient: get_auth_request con 0x7fbd600357a0 auth_method 0 2023-09-30T00:49:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.935+0000 7fbd7e59c700 1 --2- 172.21.15.6:0/70892033 >> v2:172.21.15.22:6812/142603 conn(0x7fbd600357a0 0x7fbd60037c50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009fd0 tx=0x7fbd700091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.937+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fbd7401b8c0 con 0x7fbd80098a40 2023-09-30T00:49:36.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.937+0000 7fbd6e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:36.964 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:36.937+0000 7fbd6e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:37.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd848f6700 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2023-09-30T00:49:37.183 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd848f6700 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd848f6700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbd80098170 con 0x7fbd80098a40 2023-09-30T00:49:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd6e7fc700 1 -- 172.21.15.6:0/70892033 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+227 (secure 0 0 0) 0x7fbd74024b80 con 0x7fbd80098a40 2023-09-30T00:49:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd6e7fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2023-09-30T00:49:37.184 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.183+0000 7fbd6e7fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:37.185 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:37.185 INFO:teuthology.orchestra.run.smithi006.stdout:{"status":"HEALTH_ERR","checks":{"MGR_MODULE_ERROR":{"severity":"HEALTH_ERR","summary":{"message":"Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale'","count":1},"muted":false}},"mutes":[]} 2023-09-30T00:49:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 >> v2:172.21.15.22:6812/142603 conn(0x7fbd600357a0 msgr2=0x7fbd60037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 --2- 172.21.15.6:0/70892033 >> v2:172.21.15.22:6812/142603 conn(0x7fbd600357a0 0x7fbd60037c50 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbd70009fd0 tx=0x7fbd700091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 10 monclient: shutdown 2023-09-30T00:49:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:37.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 msgr2=0x7fbd8012c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012c350 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fbd800a1df0 tx=0x7fbd74016000 comp rx=0 tx=0).stop 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 shutdown_connections 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 --2- 172.21.15.6:0/70892033 >> v2:172.21.15.22:6812/142603 conn(0x7fbd600357a0 0x7fbd60037c50 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fbd80098e90 0x7fbd8013e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fbd80098670 0x7fbd80141a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 --2- 172.21.15.6:0/70892033 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fbd80098a40 0x7fbd8012c350 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 >> 172.21.15.6:0/70892033 conn(0x7fbd800a5480 msgr2=0x7fbd80130a40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:37.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 shutdown_connections 2023-09-30T00:49:37.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.186+0000 7fbd848f6700 1 -- 172.21.15.6:0/70892033 wait complete. 2023-09-30T00:49:37.198 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2023-09-30T00:49:37.199 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:49:37.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.332+0000 7f4888285700 10 monclient: get_monmap_and_config 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.332+0000 7f4888285700 10 monclient: build_initial_monmap 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.332+0000 7f4888285700 10 monclient: monmap: 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:37.332499+0000 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:37.332499+0000 2023-09-30T00:49:37.332 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:37.333 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.332+0000 7f4888285700 1 Processor -- start 2023-09-30T00:49:37.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.332+0000 7f4888285700 1 -- start start 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient: init 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient: _add_conns ranks=[4,2,1,0,3] 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4880098670 0x7f4880132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): picked mon.noname-c con 0x7f4880098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.338 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): picked mon.noname-b con 0x7f4880098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4880098e90 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48800a1000 con 0x7f4880098e90 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48800a0a50 con 0x7f4880098670 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48800a0730 con 0x7f4880098a40 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:37.339 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4888285700 10 monclient(hunting): authenticate will time out at 1573.630794s 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40012/0 (socket says 172.21.15.6:40012) 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 1 -- 172.21.15.6:0/3555540801 learned_addr learned my addr 172.21.15.6:0/3555540801 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 10 monclient(hunting): get_auth_request con 0x7f4880098a40 auth_method 0 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.340 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.333+0000 7f4885820700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 1 --2- 172.21.15.6:0/3555540801 >> v2:172.21.15.22:3300/0 conn(0x7f4880098670 0x7f4880132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): get_auth_request con 0x7f4880098670 auth_method 0 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.341 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4886021700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 10 monclient(hunting): handle_auth_done global_id 4763 payload 274 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 1 -- 172.21.15.6:0/3555540801 >> v2:172.21.15.22:3300/0 conn(0x7f4880098670 msgr2=0x7f4880132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 1 --2- 172.21.15.6:0/3555540801 >> v2:172.21.15.22:3300/0 conn(0x7f4880098670 0x7f4880132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.342 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 1 -- 172.21.15.6:0/3555540801 >> v1:172.21.15.22:6789/0 conn(0x7f4880098e90 legacy=0x7f488012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 10 monclient: _finish_hunting 0 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 1 monclient: found mon.noname-b 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.334+0000 7f4885820700 1 -- 172.21.15.6:0/3555540801 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4880068df0 con 0x7f4880098a40 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f4885820700 1 -- 172.21.15.6:0/3555540801 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 msgr2=0x7f488012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f4885820700 1 --2- 172.21.15.6:0/3555540801 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488012ede0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f48700099b0 tx=0x7f4870009250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cc8b13394b0d1bc6 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:37.343 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 1 -- 172.21.15.6:0/3555540801 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4870009ea0 con 0x7f4880098a40 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: dump: 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.344 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: _finish_auth 0 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 1 -- 172.21.15.6:0/3555540801 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487000ba30 con 0x7f4880098a40 2023-09-30T00:49:37.345 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 1 -- 172.21.15.6:0/3555540801 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487000bdf0 con 0x7f4880098a40 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f4888285700 5 monclient: authenticate success, global_id 4763 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f4888285700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f4888285700 4 set_mon_vals no callback set 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:37.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.336+0000 7f488501f700 10 monclient: dump: 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.347 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 10 monclient: shutdown 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 -- 172.21.15.6:0/3555540801 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 msgr2=0x7f488012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 --2- 172.21.15.6:0/3555540801 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488012ede0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f48700099b0 tx=0x7f4870009250 comp rx=0 tx=0).stop 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 -- 172.21.15.6:0/3555540801 shutdown_connections 2023-09-30T00:49:37.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 --2- 172.21.15.6:0/3555540801 >> v2:172.21.15.22:3300/0 conn(0x7f4880098670 0x7f4880132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 --2- 172.21.15.6:0/3555540801 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488012ede0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 -- 172.21.15.6:0/3555540801 >> 172.21.15.6:0/3555540801 conn(0x7f48800a5480 msgr2=0x7f48800a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 -- 172.21.15.6:0/3555540801 shutdown_connections 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.337+0000 7f4888285700 1 -- 172.21.15.6:0/3555540801 wait complete. 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient: build_initial_monmap 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient: monmap: 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:37.349 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:37.350 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:37.351 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 Processor -- start 2023-09-30T00:49:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 -- start start 2023-09-30T00:49:37.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient: init 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 0x7f488012c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): picked mon.noname-a con 0x7f4880098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488009cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): picked mon.noname-c con 0x7f4880098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:37.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4880098e90 0x7f488009d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): picked mon.noname-b con 0x7f4880098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48800a1500 con 0x7f4880098670 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4880131300 con 0x7f4880098e90 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f488011c7b0 con 0x7f4880098a40 2023-09-30T00:49:37.356 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.338+0000 7f4888285700 10 monclient(hunting): authenticate will time out at 1573.635902s 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 0x7f488012c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488009cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 0x7f488012c560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35678/0 (socket says 172.21.15.6:35678) 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488009cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40020/0 (socket says 172.21.15.6:40020) 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 -- 172.21.15.6:0/3065546575 learned_addr learned my addr 172.21.15.6:0/3065546575 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:37.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): get_auth_request con 0x7f4880098670 auth_method 0 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): get_auth_request con 0x7f4880098a40 auth_method 0 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886822700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4880098e90 0x7f488009d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886822700 10 monclient(hunting): get_auth_request con 0x7f4880098e90 auth_method 0 2023-09-30T00:49:37.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886822700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886822700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886822700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient(hunting): handle_auth_done global_id 4725 payload 824 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 -- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 msgr2=0x7f488009cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488009cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 -- 172.21.15.6:0/3065546575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4880098e90 msgr2=0x7f488009d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4880098e90 0x7f488009d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient: _finish_hunting 0 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 monclient: found mon.noname-a 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4886021700 1 -- 172.21.15.6:0/3065546575 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48700094f0 con 0x7f4880098670 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.339+0000 7f4885820700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488009cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4886021700 1 -- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 msgr2=0x7f488012c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:37.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4886021700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 0x7f488012c560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f487c00ec80 tx=0x7f487c0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c00ca60 con 0x7f4880098670 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: dump: 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.362 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:37.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 5 monclient: authenticate success, global_id 4725 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f487c00cbc0 con 0x7f4880098670 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f487c013470 con 0x7f4880098670 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 10 monclient: _renew_subs 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f488011c530 con 0x7f4880098670 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f488481e700 4 set_mon_vals no callback set 2023-09-30T00:49:37.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f48767fc700 10 monclient: dump: 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.365 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 10 monclient: _renew_subs 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.341+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f488009f470 con 0x7f4880098670 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.342+0000 7f4888285700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.342+0000 7f4888285700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.342+0000 7f4888285700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.342+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f487c0137e0 con 0x7f4880098670 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.342+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48800983b0 con 0x7f4880098670 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.343+0000 7f48767fc700 1 --2- 172.21.15.6:0/3065546575 >> v2:172.21.15.22:6812/142603 conn(0x7f48640357a0 0x7f4864037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.343+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f487c047c60 con 0x7f4880098670 2023-09-30T00:49:37.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.343+0000 7f4885820700 1 --2- 172.21.15.6:0/3065546575 >> v2:172.21.15.22:6812/142603 conn(0x7f48640357a0 0x7f4864037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.343+0000 7f4885820700 10 monclient: get_auth_request con 0x7f48640357a0 auth_method 0 2023-09-30T00:49:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.346+0000 7f4885820700 1 --2- 172.21.15.6:0/3065546575 >> v2:172.21.15.22:6812/142603 conn(0x7f48640357a0 0x7f4864037c50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48700099b0 tx=0x7f4870016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.347+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f487c0238e0 con 0x7f4880098670 2023-09-30T00:49:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.347+0000 7f48767fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:37.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.347+0000 7f48767fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.552+0000 7f4888285700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:49:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.552+0000 7f4888285700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:49:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.552+0000 7f4888285700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.552+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4880072b00 con 0x7f4880098670 2023-09-30T00:49:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.553+0000 7f48767fc700 1 -- 172.21.15.6:0/3065546575 <== mon.0 v2:172.21.15.6:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+9978 (secure 0 0 0) 0x7f487c0232f0 con 0x7f4880098670 2023-09-30T00:49:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.553+0000 7f48767fc700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:49:37.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.553+0000 7f48767fc700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:37.553 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:37.553 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:29.409969+0000","last_up_change":"2023-09-30T00:44:23.223970+0000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:44:23.604669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:44:26.578400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":114301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":114301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":114301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":114301}]},"public_addr":"172.21.15.6:6804/114301","cluster_addr":"172.21.15.6:6805/114301","heartbeat_back_addr":"172.21.15.6:6807/114301","heartbeat_front_addr":"172.21.15.6:6806/114301","state":["exists","up"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":114300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":114300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":114300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":114300}]},"public_addr":"172.21.15.6:6800/114300","cluster_addr":"172.21.15.6:6801/114300","heartbeat_back_addr":"172.21.15.6:6803/114300","heartbeat_front_addr":"172.21.15.6:6802/114300","state":["exists","up"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":114302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":114302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":114302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":114302}]},"public_addr":"172.21.15.6:6808/114302","cluster_addr":"172.21.15.6:6809/114302","heartbeat_back_addr":"172.21.15.6:6811/114302","heartbeat_front_addr":"172.21.15.6:6810/114302","state":["exists","up"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":143135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6801","nonce":143135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6803","nonce":143135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":143135}]},"public_addr":"172.21.15.22:6800/143135","cluster_addr":"172.21.15.22:6801/143135","heartbeat_back_addr":"172.21.15.22:6803/143135","heartbeat_front_addr":"172.21.15.22:6802/143135","state":["exists","up"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":143134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":143134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6811","nonce":143134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":143134}]},"public_addr":"172.21.15.22:6808/143134","cluster_addr":"172.21.15.22:6809/143134","heartbeat_back_addr":"172.21.15.22:6811/143134","heartbeat_front_addr":"172.21.15.22:6810/143134","state":["exists","up"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":143133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6805","nonce":143133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6807","nonce":143133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":143133}]},"public_addr":"172.21.15.22:6804/143133","cluster_addr":"172.21.15.22:6805/143133","heartbeat_back_addr":"172.21.15.22:6807/143133","heartbeat_front_addr":"172.21.15.22:6806/143133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.597432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.528045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.614505+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.340159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.641614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.578369+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 >> v2:172.21.15.22:6812/142603 conn(0x7f48640357a0 msgr2=0x7f4864037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 --2- 172.21.15.6:0/3065546575 >> v2:172.21.15.22:6812/142603 conn(0x7f48640357a0 0x7f4864037c50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f48700099b0 tx=0x7f4870016000 comp rx=0 tx=0).stop 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 10 monclient: shutdown 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 msgr2=0x7f488012c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 0x7f488012c560 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f487c00ec80 tx=0x7f487c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 shutdown_connections 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 --2- 172.21.15.6:0/3065546575 >> v2:172.21.15.22:6812/142603 conn(0x7f48640357a0 0x7f4864037c50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4880098e90 0x7f488009d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4880098670 0x7f488012c560 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 --2- 172.21.15.6:0/3065546575 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4880098a40 0x7f488009cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 >> 172.21.15.6:0/3065546575 conn(0x7f48800a5480 msgr2=0x7f4880130a10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 shutdown_connections 2023-09-30T00:49:37.556 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.555+0000 7f4888285700 1 -- 172.21.15.6:0/3065546575 wait complete. 2023-09-30T00:49:37.567 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2023-09-30T00:49:37.568 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.698+0000 7f4c788cd700 10 monclient: get_monmap_and_config 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.698+0000 7f4c788cd700 10 monclient: build_initial_monmap 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.698+0000 7f4c788cd700 10 monclient: monmap: 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:37.698848+0000 2023-09-30T00:49:37.698 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:37.698848+0000 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:37.699 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.699+0000 7f4c788cd700 1 Processor -- start 2023-09-30T00:49:37.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.699+0000 7f4c788cd700 1 -- start start 2023-09-30T00:49:37.701 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.699+0000 7f4c788cd700 10 monclient: init 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.699+0000 7f4c788cd700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient: _add_conns ranks=[2,4,0,1,3] 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 0x7f4c74132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): picked mon.noname-b con 0x7f4c74098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4c74098a40 0x7f4c7412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): picked mon.noname-c con 0x7f4c74098a40 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f4c74098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c740a1000 con 0x7f4c74098e90 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c740a0a50 con 0x7f4c74098a40 2023-09-30T00:49:37.702 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c740a0730 con 0x7f4c74098670 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c788cd700 10 monclient(hunting): authenticate will time out at 1573.997133s 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 0x7f4c74132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c72d9d700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f4c74098e90 0x7f4c7412f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33440/0 (socket says 172.21.15.6:33440) 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c72d9d700 1 -- 172.21.15.6:0/1557033691 learned_addr learned my addr 172.21.15.6:0/1557033691 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 0x7f4c74132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40028/0 (socket says 172.21.15.6:40028) 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): get_auth_request con 0x7f4c74098670 auth_method 0 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.703 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c71d9b700 1 --2- 172.21.15.6:0/1557033691 >> v2:172.21.15.22:3300/0 conn(0x7f4c74098a40 0x7f4c7412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c71d9b700 10 monclient(hunting): get_auth_request con 0x7f4c74098a40 auth_method 0 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c71d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c71d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c71d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient(hunting): handle_auth_done global_id 4769 payload 274 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 -- 172.21.15.6:0/1557033691 >> v2:172.21.15.22:3300/0 conn(0x7f4c74098a40 msgr2=0x7f4c7412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 --2- 172.21.15.6:0/1557033691 >> v2:172.21.15.22:3300/0 conn(0x7f4c74098a40 0x7f4c7412ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 -- 172.21.15.6:0/1557033691 >> v1:172.21.15.6:6789/0 conn(0x7f4c74098e90 legacy=0x7f4c7412f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient: _finish_hunting 0 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 monclient: found mon.noname-b 2023-09-30T00:49:37.704 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:37.705 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.700+0000 7f4c7259c700 1 -- 172.21.15.6:0/1557033691 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c74068df0 con 0x7f4c74098670 2023-09-30T00:49:37.709 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.702+0000 7f4c7259c700 1 -- 172.21.15.6:0/1557033691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 msgr2=0x7f4c74132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.702+0000 7f4c7259c700 1 --2- 172.21.15.6:0/1557033691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 0x7f4c74132d00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098e0 tx=0x7f4c5c0091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5824d37452dbe33 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.702+0000 7f4c7159a700 1 -- 172.21.15.6:0/1557033691 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c009e80 con 0x7f4c74098670 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: dump: 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.710 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: _finish_auth 0 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 5 monclient: authenticate success, global_id 4769 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 1 -- 172.21.15.6:0/1557033691 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c00ba30 con 0x7f4c74098670 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:37.711 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 4 set_mon_vals no callback set 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 1 -- 172.21.15.6:0/1557033691 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c00bdb0 con 0x7f4c74098670 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c7159a700 10 monclient: dump: 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.712 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 10 monclient: shutdown 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 -- 172.21.15.6:0/1557033691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 msgr2=0x7f4c74132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 --2- 172.21.15.6:0/1557033691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 0x7f4c74132d00 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4c5c0098e0 tx=0x7f4c5c0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 -- 172.21.15.6:0/1557033691 shutdown_connections 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 --2- 172.21.15.6:0/1557033691 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098670 0x7f4c74132d00 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 --2- 172.21.15.6:0/1557033691 >> v2:172.21.15.22:3300/0 conn(0x7f4c74098a40 0x7f4c7412ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 -- 172.21.15.6:0/1557033691 >> 172.21.15.6:0/1557033691 conn(0x7f4c740a5480 msgr2=0x7f4c740a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:37.713 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 -- 172.21.15.6:0/1557033691 shutdown_connections 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.703+0000 7f4c788cd700 1 -- 172.21.15.6:0/1557033691 wait complete. 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.704+0000 7f4c788cd700 10 monclient: build_initial_monmap 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.704+0000 7f4c788cd700 10 monclient: monmap: 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:37.714 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.704+0000 7f4c788cd700 1 Processor -- start 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.704+0000 7f4c788cd700 1 -- start start 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.704+0000 7f4c788cd700 10 monclient: init 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 0x7f4c7412d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): picked mon.noname-a con 0x7f4c74098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098a40 0x7f4c7412d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): picked mon.noname-c con 0x7f4c74098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c74098e90 0x7f4c7409d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): picked mon.noname-b con 0x7f4c74098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7409d2e0 con 0x7f4c74098670 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c74131330 con 0x7f4c74098e90 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c7412edf0 con 0x7f4c74098a40 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c788cd700 10 monclient(hunting): authenticate will time out at 1574.002128s 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 0x7f4c7412d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098a40 0x7f4c7412d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 0x7f4c7412d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35690/0 (socket says 172.21.15.6:35690) 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 -- 172.21.15.6:0/3250090699 learned_addr learned my addr 172.21.15.6:0/3250090699 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:37.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): get_auth_request con 0x7f4c74098a40 auth_method 0 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): get_auth_request con 0x7f4c74098670 auth_method 0 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c74098e90 0x7f4c7409d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.717 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): get_auth_request con 0x7f4c74098e90 auth_method 0 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c72d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient(hunting): handle_auth_done global_id 4728 payload 824 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 -- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098a40 msgr2=0x7f4c7412d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.718 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098a40 0x7f4c7412d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 -- 172.21.15.6:0/3250090699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c74098e90 msgr2=0x7f4c7409d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c74098e90 0x7f4c7409d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient: _finish_hunting 0 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 monclient: found mon.noname-a 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c7259c700 1 -- 172.21.15.6:0/3250090699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c5c009420 con 0x7f4c74098670 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.705+0000 7f4c71d9b700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098a40 0x7f4c7412d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.707+0000 7f4c7259c700 1 -- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 msgr2=0x7f4c7412d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.707+0000 7f4c7259c700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 0x7f4c7412d290 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4c740a0870 tx=0x7f4c5c016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c006af0 con 0x7f4c74098670 2023-09-30T00:49:37.719 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: dump: 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c5c020070 con 0x7f4c74098670 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 5 monclient: authenticate success, global_id 4728 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 10 monclient: _renew_subs 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7412f430 con 0x7f4c74098670 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4c5c000c90 con 0x7f4c74098670 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 10 monclient: dump: 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:37.721 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c70d99700 4 set_mon_vals no callback set 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 10 monclient: _renew_subs 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c7412e7f0 con 0x7f4c74098670 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.708+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4c5c022030 con 0x7f4c74098670 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c6a7fc700 1 --2- 172.21.15.6:0/3250090699 >> v2:172.21.15.22:6812/142603 conn(0x7f4c540357a0 0x7f4c54037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c788cd700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c788cd700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c788cd700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4c5c0495d0 con 0x7f4c74098670 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c74072b00 con 0x7f4c74098670 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c71d9b700 1 --2- 172.21.15.6:0/3250090699 >> v2:172.21.15.22:6812/142603 conn(0x7f4c540357a0 0x7f4c54037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:37.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.709+0000 7f4c71d9b700 10 monclient: get_auth_request con 0x7f4c540357a0 auth_method 0 2023-09-30T00:49:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.712+0000 7f4c71d9b700 1 --2- 172.21.15.6:0/3250090699 >> v2:172.21.15.22:6812/142603 conn(0x7f4c540357a0 0x7f4c54037c50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c7412feb0 tx=0x7f4c64008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.714+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4c5c01a9c0 con 0x7f4c74098670 2023-09-30T00:49:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.714+0000 7f4c6a7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:37.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.714+0000 7f4c6a7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:37.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.914+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4c74072b00 con 0x7f4c540357a0 2023-09-30T00:49:37.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.917+0000 7f4c6a7fc700 1 -- 172.21.15.6:0/3250090699 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53819 (secure 0 0 0) 0x7f4c74072b00 con 0x7f4c540357a0 2023-09-30T00:49:37.917 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:37.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 >> v2:172.21.15.22:6812/142603 conn(0x7f4c540357a0 msgr2=0x7f4c54037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 --2- 172.21.15.6:0/3250090699 >> v2:172.21.15.22:6812/142603 conn(0x7f4c540357a0 0x7f4c54037c50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f4c7412feb0 tx=0x7f4c64008040 comp rx=0 tx=0).stop 2023-09-30T00:49:37.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 10 monclient: shutdown 2023-09-30T00:49:37.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:37.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 msgr2=0x7f4c7412d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 0x7f4c7412d290 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f4c740a0870 tx=0x7f4c5c016000 comp rx=0 tx=0).stop 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 shutdown_connections 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 --2- 172.21.15.6:0/3250090699 >> v2:172.21.15.22:6812/142603 conn(0x7f4c540357a0 0x7f4c54037c50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4c74098e90 0x7f4c7409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4c74098670 0x7f4c7412d290 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 --2- 172.21.15.6:0/3250090699 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4c74098a40 0x7f4c7412d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 >> 172.21.15.6:0/3250090699 conn(0x7f4c740a5480 msgr2=0x7f4c74131b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 shutdown_connections 2023-09-30T00:49:37.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:37.919+0000 7f4c788cd700 1 -- 172.21.15.6:0/3250090699 wait complete. 2023-09-30T00:49:37.921 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:49:37.932 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2023-09-30T00:49:37.606017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31880,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566199160,"statfs":{"total":579820584960,"available":579787939840,"internally_reserved":0,"allocated":1007616,"data_stored":1327094,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002336"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.431391+0000","last_change":"2023-09-30T00:44:28.427091+0000","last_active":"2023-09-30T00:44:28.431391+0000","last_peered":"2023-09-30T00:44:28.431391+0000","last_clean":"2023-09-30T00:44:28.431391+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:44:28.431391+0000","last_undegraded":"2023-09-30T00:44:28.431391+0000","last_fullsized":"2023-09-30T00:44:28.431391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:47:25.053141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.427410+0000","last_change":"2023-09-30T00:44:28.426953+0000","last_active":"2023-09-30T00:44:28.427410+0000","last_peered":"2023-09-30T00:44:28.427410+0000","last_clean":"2023-09-30T00:44:28.427410+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:44:28.427410+0000","last_undegraded":"2023-09-30T00:44:28.427410+0000","last_fullsized":"2023-09-30T00:44:28.427410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:21:01.653238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426484+0000","last_change":"2023-09-30T00:44:28.423208+0000","last_active":"2023-09-30T00:44:28.426484+0000","last_peered":"2023-09-30T00:44:28.426484+0000","last_clean":"2023-09-30T00:44:28.426484+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:44:28.426484+0000","last_undegraded":"2023-09-30T00:44:28.426484+0000","last_fullsized":"2023-09-30T00:44:28.426484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:04:55.556087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426680+0000","last_change":"2023-09-30T00:44:28.423004+0000","last_active":"2023-09-30T00:44:28.426680+0000","last_peered":"2023-09-30T00:44:28.426680+0000","last_clean":"2023-09-30T00:44:28.426680+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:44:28.426680+0000","last_undegraded":"2023-09-30T00:44:28.426680+0000","last_fullsized":"2023-09-30T00:44:28.426680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:22:00.541043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.417426+0000","last_change":"2023-09-30T00:44:25.575831+0000","last_active":"2023-09-30T00:44:28.417426+0000","last_peered":"2023-09-30T00:44:28.417426+0000","last_clean":"2023-09-30T00:44:28.417426+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:44:28.417426+0000","last_undegraded":"2023-09-30T00:44:28.417426+0000","last_fullsized":"2023-09-30T00:44:28.417426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_clean_scrub_stamp":"2023-09-30T00:44:24.231195+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:05.898656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424226+0000","last_change":"2023-09-30T00:44:28.422311+0000","last_active":"2023-09-30T00:44:28.424226+0000","last_peered":"2023-09-30T00:44:28.424226+0000","last_clean":"2023-09-30T00:44:28.424226+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:44:28.424226+0000","last_undegraded":"2023-09-30T00:44:28.424226+0000","last_fullsized":"2023-09-30T00:44:28.424226+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:37:58.120817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423390+0000","last_change":"2023-09-30T00:44:28.422342+0000","last_active":"2023-09-30T00:44:28.423390+0000","last_peered":"2023-09-30T00:44:28.423390+0000","last_clean":"2023-09-30T00:44:28.423390+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:44:28.423390+0000","last_undegraded":"2023-09-30T00:44:28.423390+0000","last_fullsized":"2023-09-30T00:44:28.423390+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:26.637840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426632+0000","last_change":"2023-09-30T00:44:28.424009+0000","last_active":"2023-09-30T00:44:28.426632+0000","last_peered":"2023-09-30T00:44:28.426632+0000","last_clean":"2023-09-30T00:44:28.426632+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:44:28.426632+0000","last_undegraded":"2023-09-30T00:44:28.426632+0000","last_fullsized":"2023-09-30T00:44:28.426632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:57.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424888+0000","last_change":"2023-09-30T00:44:28.424000+0000","last_active":"2023-09-30T00:44:28.424888+0000","last_peered":"2023-09-30T00:44:28.424888+0000","last_clean":"2023-09-30T00:44:28.424888+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:44:28.424888+0000","last_undegraded":"2023-09-30T00:44:28.424888+0000","last_fullsized":"2023-09-30T00:44:28.424888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:57:56.916213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.42999999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.432}]},{"osd":1,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.63300000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.496}]},{"osd":3,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.317},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.54700000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.32600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73199999999999998,"5min":2.044,"15min":2.044},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.64200000000000002},"min":{"1min":0.46300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77900000000000003,"5min":1.833,"15min":1.833},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67500000000000004,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41199999999999998,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51500000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.628,"5min":0.746,"15min":0.746},"last":0.60099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47399999999999998,"15min":0.56799999999999995},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.64200000000000002,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.49199999999999999,"15min":0.59199999999999997},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64900000000000002,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":0.65600000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65300000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.505}]},{"osd":2,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.42299999999999999},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":0.876,"15min":0.876},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45700000000000002,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.65400000000000003,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.624}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.318},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57999999999999996,"5min":1.298,"15min":1.298},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56299999999999994,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.39000000000000001,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.5,"15min":1.5},"last":0.29599999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73699999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":1.343,"15min":1.343},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71199999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.746,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.502}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78200000000000003,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77500000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.56000000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.35399999999999998,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.505,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.377,"15min":0.372},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.30499999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.435,"5min":0.37,"15min":0.37},"max":{"1min":0.65300000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.502}]},{"osd":4,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.623,"15min":0.629},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.45700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.36599999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.48399999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.41599999999999998},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.752,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.496}]},{"osd":4,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80400000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:49:37.934 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.065+0000 7f000dd16700 10 monclient: get_monmap_and_config 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.065+0000 7f000dd16700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.065+0000 7f000dd16700 10 monclient: monmap: 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:38.065878+0000 2023-09-30T00:49:38.065 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:38.065878+0000 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:38.066 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.066+0000 7f000dd16700 1 Processor -- start 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.066+0000 7f000dd16700 1 -- start start 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.066+0000 7f000dd16700 10 monclient: init 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient: _add_conns ranks=[3,4,2,0,1] 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 0x7f0008132790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): picked mon.noname-a con 0x7f0008132150 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:38.070 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f0008132e10 0x7f0008098760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): picked mon.noname-c con 0x7f0008132e10 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 0x7f0008068df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): picked mon.noname-b con 0x7f0008098ca0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0008099100 con 0x7f0008132150 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0008132520 con 0x7f0008132e10 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0008096a80 con 0x7f0008098ca0 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f000dd16700 10 monclient(hunting): authenticate will time out at 1574.364161s 2023-09-30T00:49:38.071 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 0x7f0008132790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 0x7f0008068df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 0x7f0008132790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35704/0 (socket says 172.21.15.6:35704) 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 1 -- 172.21.15.6:0/1092660476 learned_addr learned my addr 172.21.15.6:0/1092660476 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): get_auth_request con 0x7f0008132150 auth_method 0 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): get_auth_request con 0x7f0008098ca0 auth_method 0 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.072 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0006ffd700 1 --2- 172.21.15.6:0/1092660476 >> v2:172.21.15.22:3300/0 conn(0x7f0008132e10 0x7f0008098760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0006ffd700 10 monclient(hunting): get_auth_request con 0x7f0008132e10 auth_method 0 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0006ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0006ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0006ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.073 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient(hunting): handle_auth_done global_id 4775 payload 274 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 -- 172.21.15.6:0/1092660476 >> v2:172.21.15.22:3300/0 conn(0x7f0008132e10 msgr2=0x7f0008098760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 --2- 172.21.15.6:0/1092660476 >> v2:172.21.15.22:3300/0 conn(0x7f0008132e10 0x7f0008098760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 -- 172.21.15.6:0/1092660476 >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 msgr2=0x7f0008132790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 --2- 172.21.15.6:0/1092660476 >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 0x7f0008132790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 monclient: found mon.noname-b 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f0007fff700 1 -- 172.21.15.6:0/1092660476 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00080a0f80 con 0x7f0008098ca0 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.067+0000 7f00077fe700 1 --2- 172.21.15.6:0/1092660476 >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 0x7f0008132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.069+0000 7f0007fff700 1 -- 172.21.15.6:0/1092660476 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 msgr2=0x7f0008068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:38.074 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.069+0000 7f0007fff700 1 --2- 172.21.15.6:0/1092660476 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 0x7f0008068df0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f00080a1b00 tx=0x7f0000009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e8f88eb488f7e2b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 1 -- 172.21.15.6:0/1092660476 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f000000e070 con 0x7f0008098ca0 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: dump: 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.075 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 5 monclient: authenticate success, global_id 4775 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 1 -- 172.21.15.6:0/1092660476 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f000000bd70 con 0x7f0008098ca0 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 1 -- 172.21.15.6:0/1092660476 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0000012680 con 0x7f0008098ca0 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 4 set_mon_vals no callback set 2023-09-30T00:49:38.076 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f00067fc700 10 monclient: dump: 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.077 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 10 monclient: shutdown 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 -- 172.21.15.6:0/1092660476 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 msgr2=0x7f0008068df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 --2- 172.21.15.6:0/1092660476 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 0x7f0008068df0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f00080a1b00 tx=0x7f0000009780 comp rx=0 tx=0).stop 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 -- 172.21.15.6:0/1092660476 shutdown_connections 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 --2- 172.21.15.6:0/1092660476 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008098ca0 0x7f0008068df0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 --2- 172.21.15.6:0/1092660476 >> v2:172.21.15.6:3300/0 conn(0x7f0008132150 0x7f0008132790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 --2- 172.21.15.6:0/1092660476 >> v2:172.21.15.22:3300/0 conn(0x7f0008132e10 0x7f0008098760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 -- 172.21.15.6:0/1092660476 >> 172.21.15.6:0/1092660476 conn(0x7f00080a54c0 msgr2=0x7f00080a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 -- 172.21.15.6:0/1092660476 shutdown_connections 2023-09-30T00:49:38.078 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.070+0000 7f000dd16700 1 -- 172.21.15.6:0/1092660476 wait complete. 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.071+0000 7f000dd16700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.071+0000 7f000dd16700 10 monclient: monmap: 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:38.083 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.072+0000 7f000dd16700 1 Processor -- start 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.072+0000 7f000dd16700 1 -- start start 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.072+0000 7f000dd16700 10 monclient: init 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.072+0000 7f000dd16700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.072+0000 7f000dd16700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0008098ca0 0x7f000812f560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.084 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): picked mon.noname-b con 0x7f0008098ca0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 0x7f000812faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): picked mon.noname-c con 0x7f0008132150 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0008132e10 0x7f0008141920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): picked mon.noname-a con 0x7f0008132e10 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00080a1250 con 0x7f0008132e10 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f000811dce0 con 0x7f0008098ca0 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f000809f1c0 con 0x7f0008132150 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.085 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f000dd16700 10 monclient(hunting): authenticate will time out at 1574.370103s 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 0x7f000812faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 0x7f000812faa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40056/0 (socket says 172.21.15.6:40056) 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 -- 172.21.15.6:0/2662303685 learned_addr learned my addr 172.21.15.6:0/2662303685 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): get_auth_request con 0x7f0008132150 auth_method 0 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0008132e10 0x7f0008141920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): get_auth_request con 0x7f0008132e10 auth_method 0 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.086 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0007fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f00077fe700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0008098ca0 0x7f000812f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f00077fe700 10 monclient(hunting): get_auth_request con 0x7f0008098ca0 auth_method 0 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f00077fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f00077fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f00077fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient(hunting): handle_auth_done global_id 4778 payload 824 2023-09-30T00:49:38.087 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 -- 172.21.15.6:0/2662303685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0008098ca0 msgr2=0x7f000812f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0008098ca0 0x7f000812f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 -- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0008132e10 msgr2=0x7f0008141920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0008132e10 0x7f0008141920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 monclient: found mon.noname-c 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.073+0000 7f0006ffd700 1 -- 172.21.15.6:0/2662303685 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efff80094f0 con 0x7f0008132150 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f0006ffd700 1 -- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 msgr2=0x7f000812faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f0006ffd700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 0x7f000812faa0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efff000d890 tx=0x7efff0008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0012800 con 0x7f0008132150 2023-09-30T00:49:38.088 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: dump: 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.089 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efff0012960 con 0x7f0008132150 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f000dd16700 5 monclient: authenticate success, global_id 4778 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f000dd16700 10 monclient: _renew_subs 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f000dd16700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0000009a20 con 0x7f0008132150 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efff0012d00 con 0x7f0008132150 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.090 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7effff7fe700 10 monclient: dump: 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.075+0000 7f0005ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:38.091 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 10 monclient: _renew_subs 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f000811bb80 con 0x7f0008132150 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7efff0014640 con 0x7f0008132150 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.076+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0008098500 con 0x7f0008132150 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.077+0000 7effff7fe700 1 --2- 172.21.15.6:0/2662303685 >> v2:172.21.15.22:6812/142603 conn(0x7effe80357a0 0x7effe8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.077+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7efff0047ff0 con 0x7f0008132150 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.077+0000 7f00077fe700 1 --2- 172.21.15.6:0/2662303685 >> v2:172.21.15.22:6812/142603 conn(0x7effe80357a0 0x7effe8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.092 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.077+0000 7f00077fe700 10 monclient: get_auth_request con 0x7effe80357a0 auth_method 0 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.080+0000 7f00077fe700 1 --2- 172.21.15.6:0/2662303685 >> v2:172.21.15.22:6812/142603 conn(0x7effe80357a0 0x7effe8037c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efff8006fd0 tx=0x7efff8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.081+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7efff0012e60 con 0x7f0008132150 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.081+0000 7effff7fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.093 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.081+0000 7effff7fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:38.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.279+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0008098700 con 0x7effe80357a0 2023-09-30T00:49:38.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.281+0000 7effff7fe700 1 -- 172.21.15.6:0/2662303685 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53819 (secure 0 0 0) 0x7f0008098700 con 0x7effe80357a0 2023-09-30T00:49:38.281 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 >> v2:172.21.15.22:6812/142603 conn(0x7effe80357a0 msgr2=0x7effe8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 --2- 172.21.15.6:0/2662303685 >> v2:172.21.15.22:6812/142603 conn(0x7effe80357a0 0x7effe8037c50 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7efff8006fd0 tx=0x7efff8008040 comp rx=0 tx=0).stop 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 10 monclient: shutdown 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 msgr2=0x7f000812faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 0x7f000812faa0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7efff000d890 tx=0x7efff0008040 comp rx=0 tx=0).stop 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 shutdown_connections 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 --2- 172.21.15.6:0/2662303685 >> v2:172.21.15.22:6812/142603 conn(0x7effe80357a0 0x7effe8037c50 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0008132e10 0x7f0008141920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0008098ca0 0x7f000812f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 --2- 172.21.15.6:0/2662303685 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0008132150 0x7f000812faa0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 >> 172.21.15.6:0/2662303685 conn(0x7f00080a54c0 msgr2=0x7f0008097df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 shutdown_connections 2023-09-30T00:49:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.284+0000 7f000dd16700 1 -- 172.21.15.6:0/2662303685 wait complete. 2023-09-30T00:49:38.285 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:49:38.297 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2023-09-30T00:49:37.606017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31880,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566199160,"statfs":{"total":579820584960,"available":579787939840,"internally_reserved":0,"allocated":1007616,"data_stored":1327094,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002336"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.431391+0000","last_change":"2023-09-30T00:44:28.427091+0000","last_active":"2023-09-30T00:44:28.431391+0000","last_peered":"2023-09-30T00:44:28.431391+0000","last_clean":"2023-09-30T00:44:28.431391+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:44:28.431391+0000","last_undegraded":"2023-09-30T00:44:28.431391+0000","last_fullsized":"2023-09-30T00:44:28.431391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:47:25.053141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.427410+0000","last_change":"2023-09-30T00:44:28.426953+0000","last_active":"2023-09-30T00:44:28.427410+0000","last_peered":"2023-09-30T00:44:28.427410+0000","last_clean":"2023-09-30T00:44:28.427410+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:44:28.427410+0000","last_undegraded":"2023-09-30T00:44:28.427410+0000","last_fullsized":"2023-09-30T00:44:28.427410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:21:01.653238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426484+0000","last_change":"2023-09-30T00:44:28.423208+0000","last_active":"2023-09-30T00:44:28.426484+0000","last_peered":"2023-09-30T00:44:28.426484+0000","last_clean":"2023-09-30T00:44:28.426484+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:44:28.426484+0000","last_undegraded":"2023-09-30T00:44:28.426484+0000","last_fullsized":"2023-09-30T00:44:28.426484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:04:55.556087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426680+0000","last_change":"2023-09-30T00:44:28.423004+0000","last_active":"2023-09-30T00:44:28.426680+0000","last_peered":"2023-09-30T00:44:28.426680+0000","last_clean":"2023-09-30T00:44:28.426680+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:44:28.426680+0000","last_undegraded":"2023-09-30T00:44:28.426680+0000","last_fullsized":"2023-09-30T00:44:28.426680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:22:00.541043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.417426+0000","last_change":"2023-09-30T00:44:25.575831+0000","last_active":"2023-09-30T00:44:28.417426+0000","last_peered":"2023-09-30T00:44:28.417426+0000","last_clean":"2023-09-30T00:44:28.417426+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:44:28.417426+0000","last_undegraded":"2023-09-30T00:44:28.417426+0000","last_fullsized":"2023-09-30T00:44:28.417426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_clean_scrub_stamp":"2023-09-30T00:44:24.231195+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:05.898656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424226+0000","last_change":"2023-09-30T00:44:28.422311+0000","last_active":"2023-09-30T00:44:28.424226+0000","last_peered":"2023-09-30T00:44:28.424226+0000","last_clean":"2023-09-30T00:44:28.424226+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:44:28.424226+0000","last_undegraded":"2023-09-30T00:44:28.424226+0000","last_fullsized":"2023-09-30T00:44:28.424226+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:37:58.120817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423390+0000","last_change":"2023-09-30T00:44:28.422342+0000","last_active":"2023-09-30T00:44:28.423390+0000","last_peered":"2023-09-30T00:44:28.423390+0000","last_clean":"2023-09-30T00:44:28.423390+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:44:28.423390+0000","last_undegraded":"2023-09-30T00:44:28.423390+0000","last_fullsized":"2023-09-30T00:44:28.423390+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:26.637840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426632+0000","last_change":"2023-09-30T00:44:28.424009+0000","last_active":"2023-09-30T00:44:28.426632+0000","last_peered":"2023-09-30T00:44:28.426632+0000","last_clean":"2023-09-30T00:44:28.426632+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:44:28.426632+0000","last_undegraded":"2023-09-30T00:44:28.426632+0000","last_fullsized":"2023-09-30T00:44:28.426632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:57.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424888+0000","last_change":"2023-09-30T00:44:28.424000+0000","last_active":"2023-09-30T00:44:28.424888+0000","last_peered":"2023-09-30T00:44:28.424888+0000","last_clean":"2023-09-30T00:44:28.424888+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:44:28.424888+0000","last_undegraded":"2023-09-30T00:44:28.424888+0000","last_fullsized":"2023-09-30T00:44:28.424888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:57:56.916213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.42999999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.432}]},{"osd":1,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.63300000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.496}]},{"osd":3,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.317},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.54700000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.32600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73199999999999998,"5min":2.044,"15min":2.044},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.64200000000000002},"min":{"1min":0.46300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77900000000000003,"5min":1.833,"15min":1.833},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67500000000000004,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41199999999999998,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51500000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.628,"5min":0.746,"15min":0.746},"last":0.60099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47399999999999998,"15min":0.56799999999999995},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.64200000000000002,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.49199999999999999,"15min":0.59199999999999997},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64900000000000002,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":0.65600000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65300000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.505}]},{"osd":2,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.42299999999999999},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":0.876,"15min":0.876},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45700000000000002,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.65400000000000003,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.624}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.318},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57999999999999996,"5min":1.298,"15min":1.298},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56299999999999994,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.39000000000000001,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.5,"15min":1.5},"last":0.29599999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73699999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":1.343,"15min":1.343},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71199999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.746,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.502}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78200000000000003,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77500000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.56000000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.35399999999999998,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.505,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.377,"15min":0.372},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.30499999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.435,"5min":0.37,"15min":0.37},"max":{"1min":0.65300000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.502}]},{"osd":4,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.623,"15min":0.629},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.45700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.36599999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.48399999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.41599999999999998},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.752,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.496}]},{"osd":4,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80400000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:49:38.299 INFO:tasks.ceph.ceph_manager.ceph:clean! 2023-09-30T00:49:38.299 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:49:38.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.430+0000 7ff3c49ba700 10 monclient: get_monmap_and_config 2023-09-30T00:49:38.429 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.430+0000 7ff3c49ba700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.430+0000 7ff3c49ba700 10 monclient: monmap: 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:38.430293+0000 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:38.430293+0000 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.430 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:38.431 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:38.431 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:38.431 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:38.431 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:38.431 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.430+0000 7ff3c49ba700 1 Processor -- start 2023-09-30T00:49:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.430+0000 7ff3c49ba700 1 -- start start 2023-09-30T00:49:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.430+0000 7ff3c49ba700 10 monclient: init 2023-09-30T00:49:38.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient: _add_conns ranks=[4,2,0,1,3] 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7ff3c0098670 0x7ff3c0132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): picked mon.noname-c con 0x7ff3c0098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 0x7ff3c012ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): picked mon.noname-b con 0x7ff3c0098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7ff3c0098e90 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c00a1000 con 0x7ff3c0098e90 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c00a0a50 con 0x7ff3c0098670 2023-09-30T00:49:38.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00a0730 con 0x7ff3c0098a40 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3c49ba700 10 monclient(hunting): authenticate will time out at 1574.728485s 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 0x7ff3c012ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bed9d700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7ff3c0098e90 0x7ff3c012f320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33448/0 (socket says 172.21.15.6:33448) 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 0x7ff3c012ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40068/0 (socket says 172.21.15.6:40068) 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bed9d700 1 -- 172.21.15.6:0/838473843 learned_addr learned my addr 172.21.15.6:0/838473843 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): get_auth_request con 0x7ff3c0098a40 auth_method 0 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 1 --2- 172.21.15.6:0/838473843 >> v2:172.21.15.22:3300/0 conn(0x7ff3c0098670 0x7ff3c0132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): get_auth_request con 0x7ff3c0098670 auth_method 0 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3be59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.440 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient(hunting): handle_auth_done global_id 4781 payload 274 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 1 -- 172.21.15.6:0/838473843 >> v2:172.21.15.22:3300/0 conn(0x7ff3c0098670 msgr2=0x7ff3c0132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 1 --2- 172.21.15.6:0/838473843 >> v2:172.21.15.22:3300/0 conn(0x7ff3c0098670 0x7ff3c0132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 1 -- 172.21.15.6:0/838473843 >> v1:172.21.15.6:6789/0 conn(0x7ff3c0098e90 legacy=0x7ff3c012f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.431+0000 7ff3bdd9b700 1 monclient: found mon.noname-b 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.432+0000 7ff3bdd9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.432+0000 7ff3bdd9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.432+0000 7ff3bdd9b700 1 -- 172.21.15.6:0/838473843 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c0068df0 con 0x7ff3c0098a40 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bdd9b700 1 -- 172.21.15.6:0/838473843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 msgr2=0x7ff3c012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bdd9b700 1 --2- 172.21.15.6:0/838473843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 0x7ff3c012ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80098e0 tx=0x7ff3a80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae0b494295c2f470 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 1 -- 172.21.15.6:0/838473843 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8011840 con 0x7ff3c0098a40 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.441 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: dump: 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.442 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 1 -- 172.21.15.6:0/838473843 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3a80119a0 con 0x7ff3c0098a40 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 5 monclient: authenticate success, global_id 4781 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 1 -- 172.21.15.6:0/838473843 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8011d40 con 0x7ff3c0098a40 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 4 set_mon_vals no callback set 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3bd59a700 10 monclient: dump: 2023-09-30T00:49:38.443 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 10 monclient: shutdown 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 1 -- 172.21.15.6:0/838473843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 msgr2=0x7ff3c012ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.434+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/838473843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 0x7ff3c012ede0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80098e0 tx=0x7ff3a80091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 -- 172.21.15.6:0/838473843 shutdown_connections 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/838473843 >> v2:172.21.15.22:3300/0 conn(0x7ff3c0098670 0x7ff3c0132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/838473843 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098a40 0x7ff3c012ede0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 -- 172.21.15.6:0/838473843 >> 172.21.15.6:0/838473843 conn(0x7ff3c00a5480 msgr2=0x7ff3c00a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 -- 172.21.15.6:0/838473843 shutdown_connections 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 -- 172.21.15.6:0/838473843 wait complete. 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 10 monclient: monmap: 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:38.445 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.435+0000 7ff3c49ba700 1 Processor -- start 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 -- start start 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient: init 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:38.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 0x7ff3c01418c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): picked mon.noname-c con 0x7ff3c0098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff3c0098a40 0x7ff3c012c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): picked mon.noname-b con 0x7ff3c0098a40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 0x7ff3c012c740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): picked mon.noname-a con 0x7ff3c0098e90 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c009f3e0 con 0x7ff3c0098e90 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c01313c0 con 0x7ff3c0098a40 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c00a1500 con 0x7ff3c0098670 2023-09-30T00:49:38.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3c49ba700 10 monclient(hunting): authenticate will time out at 1574.733541s 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 0x7ff3c01418c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 0x7ff3c012c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 0x7ff3c01418c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40070/0 (socket says 172.21.15.6:40070) 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 0x7ff3c012c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35712/0 (socket says 172.21.15.6:35712) 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 1 -- 172.21.15.6:0/1104012962 learned_addr learned my addr 172.21.15.6:0/1104012962 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): get_auth_request con 0x7ff3c0098670 auth_method 0 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): get_auth_request con 0x7ff3c0098e90 auth_method 0 2023-09-30T00:49:38.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bdd9b700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff3c0098a40 0x7ff3c012c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bdd9b700 10 monclient(hunting): get_auth_request con 0x7ff3c0098a40 auth_method 0 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bdd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bdd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3bdd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 10 monclient(hunting): handle_auth_done global_id 4784 payload 824 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 1 -- 172.21.15.6:0/1104012962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff3c0098a40 msgr2=0x7ff3c012c200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.436+0000 7ff3be59c700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff3c0098a40 0x7ff3c012c200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 1 -- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 msgr2=0x7ff3c012c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 0x7ff3c012c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 1 monclient: found mon.noname-c 2023-09-30T00:49:38.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3be59c700 1 -- 172.21.15.6:0/1104012962 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3a8009420 con 0x7ff3c0098670 2023-09-30T00:49:38.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.437+0000 7ff3bed9d700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 0x7ff3c012c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:38.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3be59c700 1 -- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 msgr2=0x7ff3c01418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3be59c700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 0x7ff3c01418c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000ece0 tx=0x7ff3b00091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b000cb30 con 0x7ff3c0098670 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: dump: 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.453 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3b000cc90 con 0x7ff3c0098670 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 5 monclient: authenticate success, global_id 4784 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 10 monclient: _renew_subs 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3c011c350 con 0x7ff3c0098670 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b0013470 con 0x7ff3c0098670 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3b67fc700 10 monclient: dump: 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.455 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3bcd99700 4 set_mon_vals no callback set 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 10 monclient: _renew_subs 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.439+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c009e6c0 con 0x7ff3c0098670 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff3b0013820 con 0x7ff3c0098670 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3b67fc700 1 --2- 172.21.15.6:0/1104012962 >> v2:172.21.15.22:6812/142603 conn(0x7ff3a00357a0 0x7ff3a0037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff3b0047cd0 con 0x7ff3c0098670 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3c49ba700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3c49ba700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3c49ba700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c0072b00 con 0x7ff3c0098670 2023-09-30T00:49:38.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3bdd9b700 1 --2- 172.21.15.6:0/1104012962 >> v2:172.21.15.22:6812/142603 conn(0x7ff3a00357a0 0x7ff3a0037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.440+0000 7ff3bdd9b700 10 monclient: get_auth_request con 0x7ff3a00357a0 auth_method 0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.443+0000 7ff3bdd9b700 1 --2- 172.21.15.6:0/1104012962 >> v2:172.21.15.22:6812/142603 conn(0x7ff3a00357a0 0x7ff3a0037c50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80030c0 tx=0x7ff3a8016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.445+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff3b00238e0 con 0x7ff3c0098670 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.445+0000 7ff3b67fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.445+0000 7ff3b67fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:38.638 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.639+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff3c0063920 con 0x7ff3a00357a0 2023-09-30T00:49:38.641 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.641+0000 7ff3b67fc700 1 -- 172.21.15.6:0/1104012962 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53819 (secure 0 0 0) 0x7ff3c0063920 con 0x7ff3a00357a0 2023-09-30T00:49:38.642 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 >> v2:172.21.15.22:6812/142603 conn(0x7ff3a00357a0 msgr2=0x7ff3a0037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/1104012962 >> v2:172.21.15.22:6812/142603 conn(0x7ff3a00357a0 0x7ff3a0037c50 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff3a80030c0 tx=0x7ff3a8016000 comp rx=0 tx=0).stop 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 10 monclient: shutdown 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 msgr2=0x7ff3c01418c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 0x7ff3c01418c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff3b000ece0 tx=0x7ff3b00091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 shutdown_connections 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/1104012962 >> v2:172.21.15.22:6812/142603 conn(0x7ff3a00357a0 0x7ff3a0037c50 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.644 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff3c0098e90 0x7ff3c012c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff3c0098670 0x7ff3c01418c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 --2- 172.21.15.6:0/1104012962 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff3c0098a40 0x7ff3c012c200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 >> 172.21.15.6:0/1104012962 conn(0x7ff3c00a5480 msgr2=0x7ff3c0130920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 shutdown_connections 2023-09-30T00:49:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.644+0000 7ff3c49ba700 1 -- 172.21.15.6:0/1104012962 wait complete. 2023-09-30T00:49:38.645 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:49:38.656 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":169,"stamp":"2023-09-30T00:49:37.606017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31880,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566199160,"statfs":{"total":579820584960,"available":579787939840,"internally_reserved":0,"allocated":1007616,"data_stored":1327094,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002336"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.431391+0000","last_change":"2023-09-30T00:44:28.427091+0000","last_active":"2023-09-30T00:44:28.431391+0000","last_peered":"2023-09-30T00:44:28.431391+0000","last_clean":"2023-09-30T00:44:28.431391+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:44:28.431391+0000","last_undegraded":"2023-09-30T00:44:28.431391+0000","last_fullsized":"2023-09-30T00:44:28.431391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:47:25.053141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.427410+0000","last_change":"2023-09-30T00:44:28.426953+0000","last_active":"2023-09-30T00:44:28.427410+0000","last_peered":"2023-09-30T00:44:28.427410+0000","last_clean":"2023-09-30T00:44:28.427410+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:44:28.427410+0000","last_undegraded":"2023-09-30T00:44:28.427410+0000","last_fullsized":"2023-09-30T00:44:28.427410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:21:01.653238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426484+0000","last_change":"2023-09-30T00:44:28.423208+0000","last_active":"2023-09-30T00:44:28.426484+0000","last_peered":"2023-09-30T00:44:28.426484+0000","last_clean":"2023-09-30T00:44:28.426484+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:44:28.426484+0000","last_undegraded":"2023-09-30T00:44:28.426484+0000","last_fullsized":"2023-09-30T00:44:28.426484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:04:55.556087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426680+0000","last_change":"2023-09-30T00:44:28.423004+0000","last_active":"2023-09-30T00:44:28.426680+0000","last_peered":"2023-09-30T00:44:28.426680+0000","last_clean":"2023-09-30T00:44:28.426680+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:44:28.426680+0000","last_undegraded":"2023-09-30T00:44:28.426680+0000","last_fullsized":"2023-09-30T00:44:28.426680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:22:00.541043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.417426+0000","last_change":"2023-09-30T00:44:25.575831+0000","last_active":"2023-09-30T00:44:28.417426+0000","last_peered":"2023-09-30T00:44:28.417426+0000","last_clean":"2023-09-30T00:44:28.417426+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:44:28.417426+0000","last_undegraded":"2023-09-30T00:44:28.417426+0000","last_fullsized":"2023-09-30T00:44:28.417426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_clean_scrub_stamp":"2023-09-30T00:44:24.231195+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:05.898656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424226+0000","last_change":"2023-09-30T00:44:28.422311+0000","last_active":"2023-09-30T00:44:28.424226+0000","last_peered":"2023-09-30T00:44:28.424226+0000","last_clean":"2023-09-30T00:44:28.424226+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:44:28.424226+0000","last_undegraded":"2023-09-30T00:44:28.424226+0000","last_fullsized":"2023-09-30T00:44:28.424226+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:37:58.120817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423390+0000","last_change":"2023-09-30T00:44:28.422342+0000","last_active":"2023-09-30T00:44:28.423390+0000","last_peered":"2023-09-30T00:44:28.423390+0000","last_clean":"2023-09-30T00:44:28.423390+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:44:28.423390+0000","last_undegraded":"2023-09-30T00:44:28.423390+0000","last_fullsized":"2023-09-30T00:44:28.423390+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:26.637840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426632+0000","last_change":"2023-09-30T00:44:28.424009+0000","last_active":"2023-09-30T00:44:28.426632+0000","last_peered":"2023-09-30T00:44:28.426632+0000","last_clean":"2023-09-30T00:44:28.426632+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:44:28.426632+0000","last_undegraded":"2023-09-30T00:44:28.426632+0000","last_fullsized":"2023-09-30T00:44:28.426632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:57.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424888+0000","last_change":"2023-09-30T00:44:28.424000+0000","last_active":"2023-09-30T00:44:28.424888+0000","last_peered":"2023-09-30T00:44:28.424888+0000","last_clean":"2023-09-30T00:44:28.424888+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:44:28.424888+0000","last_undegraded":"2023-09-30T00:44:28.424888+0000","last_fullsized":"2023-09-30T00:44:28.424888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:57:56.916213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.42999999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.432}]},{"osd":1,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.63300000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.496}]},{"osd":3,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.317},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.54700000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.32600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73199999999999998,"5min":2.044,"15min":2.044},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.64200000000000002},"min":{"1min":0.46300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77900000000000003,"5min":1.833,"15min":1.833},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67500000000000004,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41199999999999998,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51500000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.628,"5min":0.746,"15min":0.746},"last":0.60099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47399999999999998,"15min":0.56799999999999995},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.64200000000000002,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.49199999999999999,"15min":0.59199999999999997},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64900000000000002,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":0.65600000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640320,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65300000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.505}]},{"osd":2,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.42299999999999999},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":0.876,"15min":0.876},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45700000000000002,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.65400000000000003,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.624}]}]},{"osd":2,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.318},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57999999999999996,"5min":1.298,"15min":1.298},"last":0.34000000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56299999999999994,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.39000000000000001,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.5,"15min":1.5},"last":0.29599999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73699999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":1.343,"15min":1.343},"last":0.60199999999999998}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71199999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.746,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.502}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78200000000000003,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77500000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.56000000000000005}]}]},{"osd":0,"up_from":11,"seq":47244640320,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.35399999999999998,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.505,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.377,"15min":0.372},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.30499999999999999}]},{"osd":3,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.435,"5min":0.37,"15min":0.37},"max":{"1min":0.65300000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.502}]},{"osd":4,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.627,"5min":0.623,"15min":0.629},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.55600000000000005}]},{"osd":5,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.53800000000000003}]}]},{"osd":1,"up_from":11,"seq":47244640320,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33900000000000002,"15min":0.34000000000000002},"min":{"1min":0.26200000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.45700000000000002,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.373,"5min":0.36599999999999999,"15min":0.371},"min":{"1min":0.27800000000000002,"5min":0.248,"15min":0.248},"max":{"1min":0.48399999999999999,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.32500000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38300000000000001,"15min":0.38300000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.40999999999999998,"15min":0.41599999999999998},"min":{"1min":0.33700000000000002,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.34799999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54300000000000004,"15min":0.55500000000000005},"min":{"1min":0.40699999999999997,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.752,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.55400000000000005,"15min":0.55000000000000004},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.72599999999999998,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.496}]},{"osd":4,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.76900000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.61799999999999999,"15min":0.626},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.80400000000000005,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:48:34 2023","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.56599999999999995,"15min":0.56799999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78300000000000003,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.439},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.58299999999999996,"15min":0.59099999999999997},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.77600000000000002,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.40400000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:49:38.658 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2023-09-30T00:49:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.791+0000 7f8ae1e36700 10 monclient: get_monmap_and_config 2023-09-30T00:49:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.791+0000 7f8ae1e36700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.791+0000 7f8ae1e36700 10 monclient: monmap: 2023-09-30T00:49:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:38.791 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:38.792042+0000 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:38.792042+0000 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:38.792 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.794 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.792+0000 7f8ae1e36700 1 Processor -- start 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.792+0000 7f8ae1e36700 1 -- start start 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.792+0000 7f8ae1e36700 10 monclient: init 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient: _add_conns ranks=[3,0,2,4,1] 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8adc098670 0x7f8adc132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): picked mon.noname-a con 0x7f8adc098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f8adc098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 0x7f8adc12f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): picked mon.noname-b con 0x7f8adc098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8adc0a1000 con 0x7f8adc098a40 2023-09-30T00:49:38.795 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8adc0a0a50 con 0x7f8adc098670 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0a0730 con 0x7f8adc098e90 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8ae1e36700 10 monclient(hunting): authenticate will time out at 1575.090273s 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f8adc098670 0x7f8adc132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 0x7f8adc12f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adaffd700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f8adc098a40 0x7f8adc12ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33456/0 (socket says 172.21.15.6:33456) 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adaffd700 1 -- 172.21.15.6:0/1393159631 learned_addr learned my addr 172.21.15.6:0/1393159631 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): get_auth_request con 0x7f8adc098670 auth_method 0 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.796 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): get_auth_request con 0x7f8adc098e90 auth_method 0 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient(hunting): handle_auth_done global_id 4787 payload 274 2023-09-30T00:49:38.797 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 1 -- 172.21.15.6:0/1393159631 >> v2:172.21.15.6:3300/0 conn(0x7f8adc098670 msgr2=0x7f8adc132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 1 --2- 172.21.15.6:0/1393159631 >> v2:172.21.15.6:3300/0 conn(0x7f8adc098670 0x7f8adc132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 1 -- 172.21.15.6:0/1393159631 >> v1:172.21.15.6:6789/0 conn(0x7f8adc098a40 legacy=0x7f8adc12ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 1 monclient: found mon.noname-b 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adbfff700 1 -- 172.21.15.6:0/1393159631 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8adc068df0 con 0x7f8adc098e90 2023-09-30T00:49:38.798 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.793+0000 7f8adb7fe700 1 --2- 172.21.15.6:0/1393159631 >> v2:172.21.15.6:3300/0 conn(0x7f8adc098670 0x7f8adc132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8adbfff700 1 -- 172.21.15.6:0/1393159631 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 msgr2=0x7f8adc12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8adbfff700 1 --2- 172.21.15.6:0/1393159631 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 0x7f8adc12f4f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009e60 tx=0x7f8ad4009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dbbde242afaf2242 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 1 -- 172.21.15.6:0/1393159631 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400e070 con 0x7f8adc098e90 2023-09-30T00:49:38.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: dump: 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.804 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 5 monclient: authenticate success, global_id 4787 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 1 -- 172.21.15.6:0/1393159631 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ad400bd30 con 0x7f8adc098e90 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 4 set_mon_vals no callback set 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 1 -- 172.21.15.6:0/1393159631 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad400bea0 con 0x7f8adc098e90 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:38.805 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ada7fc700 10 monclient: dump: 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 10 monclient: shutdown 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 -- 172.21.15.6:0/1393159631 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 msgr2=0x7f8adc12f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/1393159631 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 0x7f8adc12f4f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ad4009e60 tx=0x7f8ad4009780 comp rx=0 tx=0).stop 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 -- 172.21.15.6:0/1393159631 shutdown_connections 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/1393159631 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098e90 0x7f8adc12f4f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/1393159631 >> v2:172.21.15.6:3300/0 conn(0x7f8adc098670 0x7f8adc132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 -- 172.21.15.6:0/1393159631 >> 172.21.15.6:0/1393159631 conn(0x7f8adc0a5480 msgr2=0x7f8adc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 -- 172.21.15.6:0/1393159631 shutdown_connections 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.796+0000 7f8ae1e36700 1 -- 172.21.15.6:0/1393159631 wait complete. 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.797+0000 7f8ae1e36700 10 monclient: build_initial_monmap 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.797+0000 7f8ae1e36700 10 monclient: monmap: 2023-09-30T00:49:38.807 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.797+0000 7f8ae1e36700 1 Processor -- start 2023-09-30T00:49:38.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 -- start start 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient: init 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8adc098670 0x7f8adc12d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): picked mon.noname-a con 0x7f8adc098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 0x7f8adc12d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): picked mon.noname-c con 0x7f8adc098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8adc098e90 0x7f8adc09d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): picked mon.noname-b con 0x7f8adc098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc0a1500 con 0x7f8adc098670 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc09d220 con 0x7f8adc098e90 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8adc12ed40 con 0x7f8adc098a40 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8ae1e36700 10 monclient(hunting): authenticate will time out at 1575.095529s 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 0x7f8adc12d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 0x7f8adc12d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40090/0 (socket says 172.21.15.6:40090) 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 -- 172.21.15.6:0/2828744214 learned_addr learned my addr 172.21.15.6:0/2828744214 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): get_auth_request con 0x7f8adc098a40 auth_method 0 2023-09-30T00:49:38.810 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8adc098670 0x7f8adc12d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): get_auth_request con 0x7f8adc098670 auth_method 0 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:38.811 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adbfff700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8adc098e90 0x7f8adc09d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adbfff700 10 monclient(hunting): get_auth_request con 0x7f8adc098e90 auth_method 0 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adbfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adbfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adbfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient(hunting): handle_auth_done global_id 4790 payload 824 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 -- 172.21.15.6:0/2828744214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8adc098e90 msgr2=0x7f8adc09d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8adc098e90 0x7f8adc09d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 -- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8adc098670 msgr2=0x7f8adc12d2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:38.812 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8adc098670 0x7f8adc12d2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.798+0000 7f8adaffd700 1 monclient: found mon.noname-c 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.799+0000 7f8adaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.799+0000 7f8adaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.799+0000 7f8adaffd700 1 -- 172.21.15.6:0/2828744214 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8acc009420 con 0x7f8adc098a40 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8adaffd700 1 -- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 msgr2=0x7f8adc12d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8adaffd700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 0x7f8adc12d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c5e0 tx=0x7f8ac4004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4005d40 con 0x7f8adc098a40 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: dump: 2023-09-30T00:49:38.813 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:38.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ac40070d0 con 0x7f8adc098a40 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 5 monclient: authenticate success, global_id 4790 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 10 monclient: _renew_subs 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad4009a20 con 0x7f8adc098a40 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4007490 con 0x7f8adc098a40 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad37fe700 10 monclient: dump: 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:38.815 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 10 monclient: _renew_subs 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ad9ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.801+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8adc12f400 con 0x7f8adc098a40 2023-09-30T00:49:38.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f8ac401b3b0 con 0x7f8adc098a40 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ae1e36700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ae1e36700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ae1e36700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8adc09cea0 con 0x7f8adc098a40 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ad37fe700 1 --2- 172.21.15.6:0/2828744214 >> v2:172.21.15.22:6812/142603 conn(0x7f8abc0357f0 0x7f8abc037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.802+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f8ac4046f40 con 0x7f8adc098a40 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.803+0000 7f8adb7fe700 1 --2- 172.21.15.6:0/2828744214 >> v2:172.21.15.22:6812/142603 conn(0x7f8abc0357f0 0x7f8abc037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:38.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.803+0000 7f8adb7fe700 10 monclient: get_auth_request con 0x7f8abc0357f0 auth_method 0 2023-09-30T00:49:38.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.805+0000 7f8adb7fe700 1 --2- 172.21.15.6:0/2828744214 >> v2:172.21.15.22:6812/142603 conn(0x7f8abc0357f0 0x7f8abc037ca0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009f40 tx=0x7f8acc0091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:38.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.807+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f8adc09cea0 con 0x7f8adc098a40 2023-09-30T00:49:38.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.807+0000 7f8ad37fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:38.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:38.807+0000 7f8ad37fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.007+0000 7f8ae1e36700 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:49:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.007+0000 7f8ae1e36700 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:49:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.007+0000 7f8ae1e36700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:39.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.007+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8adc09cea0 con 0x7f8adc098a40 2023-09-30T00:49:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.008+0000 7f8ad37fe700 1 -- 172.21.15.6:0/2828744214 <== mon.2 v2:172.21.15.6:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+9978 (secure 0 0 0) 0x7f8adc09cea0 con 0x7f8adc098a40 2023-09-30T00:49:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.008+0000 7f8ad37fe700 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2023-09-30T00:49:39.008 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.008+0000 7f8ad37fe700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:49:39.008 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:39.008 INFO:teuthology.orchestra.run.smithi006.stdout:{"epoch":17,"fsid":"cb933378-6423-4c15-869d-4727cefb664a","created":"2023-09-30T00:44:13.353775+0000","modified":"2023-09-30T00:44:29.409969+0000","last_up_change":"2023-09-30T00:44:23.223970+0000","last_in_change":"2023-09-30T00:44:16.449558+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-09-30T00:44:23.604669+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"rbd","create_time":"2023-09-30T00:44:26.578400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"97153c5d-1a02-4a71-82e1-ab2dc690ab09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6804","nonce":114301}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6805","nonce":114301}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6807","nonce":114301}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6806","nonce":114301}]},"public_addr":"172.21.15.6:6804/114301","cluster_addr":"172.21.15.6:6805/114301","heartbeat_back_addr":"172.21.15.6:6807/114301","heartbeat_front_addr":"172.21.15.6:6806/114301","state":["exists","up"]},{"osd":1,"uuid":"7a70cc7a-56b4-4c35-a0a7-0fad04dbb66d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6800","nonce":114300}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6801","nonce":114300}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6803","nonce":114300}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6802","nonce":114300}]},"public_addr":"172.21.15.6:6800/114300","cluster_addr":"172.21.15.6:6801/114300","heartbeat_back_addr":"172.21.15.6:6803/114300","heartbeat_front_addr":"172.21.15.6:6802/114300","state":["exists","up"]},{"osd":2,"uuid":"6281fb09-a2a6-427f-9ae1-f8e4d0a6bfee","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6808","nonce":114302}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6809","nonce":114302}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6811","nonce":114302}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.6:6810","nonce":114302}]},"public_addr":"172.21.15.6:6808/114302","cluster_addr":"172.21.15.6:6809/114302","heartbeat_back_addr":"172.21.15.6:6811/114302","heartbeat_front_addr":"172.21.15.6:6810/114302","state":["exists","up"]},{"osd":3,"uuid":"f0be84e5-ee32-4470-9d79-7c4942bae02b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6800","nonce":143135}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6801","nonce":143135}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6803","nonce":143135}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6802","nonce":143135}]},"public_addr":"172.21.15.22:6800/143135","cluster_addr":"172.21.15.22:6801/143135","heartbeat_back_addr":"172.21.15.22:6803/143135","heartbeat_front_addr":"172.21.15.22:6802/143135","state":["exists","up"]},{"osd":4,"uuid":"061ce3a2-0e9b-448e-9cae-789937c85eab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6808","nonce":143134}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6809","nonce":143134}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6811","nonce":143134}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6810","nonce":143134}]},"public_addr":"172.21.15.22:6808/143134","cluster_addr":"172.21.15.22:6809/143134","heartbeat_back_addr":"172.21.15.22:6811/143134","heartbeat_front_addr":"172.21.15.22:6810/143134","state":["exists","up"]},{"osd":5,"uuid":"28f01a16-aeec-4d18-942d-a6a37dbbfe77","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6804","nonce":143133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6805","nonce":143133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6807","nonce":143133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.22:6806","nonce":143133}]},"public_addr":"172.21.15.22:6804/143133","cluster_addr":"172.21.15.22:6805/143133","heartbeat_back_addr":"172.21.15.22:6807/143133","heartbeat_front_addr":"172.21.15.22:6806/143133","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.597432+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.528045+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.614505+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.340159+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.641614+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2023-09-30T00:44:21.578369+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-09-30T00:49:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.010+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 >> v2:172.21.15.22:6812/142603 conn(0x7f8abc0357f0 msgr2=0x7f8abc037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:39.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.010+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/2828744214 >> v2:172.21.15.22:6812/142603 conn(0x7f8abc0357f0 0x7f8abc037ca0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f8acc009f40 tx=0x7f8acc0091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.010+0000 7f8ae1e36700 10 monclient: shutdown 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.010+0000 7f8ae1e36700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.010+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 msgr2=0x7f8adc12d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.010+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 0x7f8adc12d810 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c5e0 tx=0x7f8ac4004a80 comp rx=0 tx=0).stop 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 shutdown_connections 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/2828744214 >> v2:172.21.15.22:6812/142603 conn(0x7f8abc0357f0 0x7f8abc037ca0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f8adc098e90 0x7f8adc09d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f8adc098670 0x7f8adc12d2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 --2- 172.21.15.6:0/2828744214 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f8adc098a40 0x7f8adc12d810 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:39.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 >> 172.21.15.6:0/2828744214 conn(0x7f8adc0a5480 msgr2=0x7f8adc069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 shutdown_connections 2023-09-30T00:49:39.012 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:39.011+0000 7f8ae1e36700 1 -- 172.21.15.6:0/2828744214 wait complete. 2023-09-30T00:49:40.024 INFO:tasks.ceph:Scrubbing osd.0 2023-09-30T00:49:40.025 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:49:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.169+0000 7f3ece884700 10 monclient: get_monmap_and_config 2023-09-30T00:49:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.169+0000 7f3ece884700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.169 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.169+0000 7f3ece884700 10 monclient: monmap: 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:40.170081+0000 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:40.170081+0000 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:40.170 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:40.171 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.170+0000 7f3ece884700 1 Processor -- start 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.170+0000 7f3ece884700 1 -- start start 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.170+0000 7f3ece884700 10 monclient: init 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient: _add_conns ranks=[3,0,1,2,4] 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ec81322a0 0x7f3ec8098670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient(hunting): picked mon.noname-a con 0x7f3ec81322a0 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:40.174 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f3ec8132e00 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f3ec8098bb0 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec8132670 con 0x7f3ec8132e00 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec812fbb0 con 0x7f3ec8098bb0 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80a1160 con 0x7f3ec81322a0 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ece884700 10 monclient(hunting): authenticate will time out at 1576.468260s 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ec81322a0 0x7f3ec8098670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f3ec81322a0 0x7f3ec8098670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35742/0 (socket says 172.21.15.6:35742) 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1074618720 learned_addr learned my addr 172.21.15.6:0/1074618720 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:40.175 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): get_auth_request con 0x7f3ec81322a0 auth_method 0 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec77fe700 1 --1- 172.21.15.6:0/1074618720 >> v1:172.21.15.6:6789/0 conn(0x7f3ec8132e00 0x7f3ec812ed90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33460/0 (socket says 172.21.15.6:33460) 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_done global_id 4746 payload 274 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1074618720 >> v1:172.21.15.22:6789/0 conn(0x7f3ec8098bb0 legacy=0x7f3ec812f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1074618720 >> v1:172.21.15.6:6789/0 conn(0x7f3ec8132e00 legacy=0x7f3ec812ed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.176 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 monclient: found mon.noname-a 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.171+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1074618720 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8068f00 con 0x7f3ec81322a0 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.173+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1074618720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec81322a0 msgr2=0x7f3ec8098670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.173+0000 7f3ec7fff700 1 --2- 172.21.15.6:0/1074618720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec81322a0 0x7f3ec8098670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80a12a0 tx=0x7f3eb8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77002f02fe1de22d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 1 -- 172.21.15.6:0/1074618720 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb800ba30 con 0x7f3ec81322a0 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.177 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: dump: 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.178 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 1 -- 172.21.15.6:0/1074618720 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb800bb90 con 0x7f3ec81322a0 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 5 monclient: authenticate success, global_id 4746 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:40.179 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 1 -- 172.21.15.6:0/1074618720 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8011420 con 0x7f3ec81322a0 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 4 set_mon_vals no callback set 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ec6ffd700 10 monclient: dump: 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.180 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 10 monclient: shutdown 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 -- 172.21.15.6:0/1074618720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec81322a0 msgr2=0x7f3ec8098670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 --2- 172.21.15.6:0/1074618720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec81322a0 0x7f3ec8098670 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3ec80a12a0 tx=0x7f3eb8009250 comp rx=0 tx=0).stop 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 -- 172.21.15.6:0/1074618720 shutdown_connections 2023-09-30T00:49:40.181 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 --2- 172.21.15.6:0/1074618720 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec81322a0 0x7f3ec8098670 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 -- 172.21.15.6:0/1074618720 >> 172.21.15.6:0/1074618720 conn(0x7f3ec80a54c0 msgr2=0x7f3ec80a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 -- 172.21.15.6:0/1074618720 shutdown_connections 2023-09-30T00:49:40.182 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.174+0000 7f3ece884700 1 -- 172.21.15.6:0/1074618720 wait complete. 2023-09-30T00:49:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.175+0000 7f3ece884700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.175+0000 7f3ece884700 10 monclient: monmap: 2023-09-30T00:49:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.187 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.176+0000 7f3ece884700 1 Processor -- start 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.176+0000 7f3ece884700 1 -- start start 2023-09-30T00:49:40.188 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.176+0000 7f3ece884700 10 monclient: init 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.176+0000 7f3ece884700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.176+0000 7f3ece884700 10 monclient: _add_conns ranks=[0,1,2] 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.176+0000 7f3ece884700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 0x7f3ec809eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): picked mon.noname-a con 0x7f3ec8098bb0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ec81322a0 0x7f3ec812e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): picked mon.noname-b con 0x7f3ec81322a0 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3ec8132e00 0x7f3ec81418e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): picked mon.noname-c con 0x7f3ec8132e00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80a1540 con 0x7f3ec8098bb0 2023-09-30T00:49:40.189 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80a4650 con 0x7f3ec81322a0 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec811e4a0 con 0x7f3ec8132e00 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ece884700 10 monclient(hunting): authenticate will time out at 1576.474085s 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 0x7f3ec809eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 0x7f3ec809eac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35752/0 (socket says 172.21.15.6:35752) 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1254000937 learned_addr learned my addr 172.21.15.6:0/1254000937 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): get_auth_request con 0x7f3ec8098bb0 auth_method 0 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.190 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 1 --2- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3ec8132e00 0x7f3ec81418e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): get_auth_request con 0x7f3ec8132e00 auth_method 0 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.191 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec77fe700 1 --2- 172.21.15.6:0/1254000937 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ec81322a0 0x7f3ec812e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ecce21700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec77fe700 10 monclient(hunting): get_auth_request con 0x7f3ec81322a0 auth_method 0 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient(hunting): handle_auth_done global_id 4749 payload 824 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3ec8132e00 msgr2=0x7f3ec81418e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 --2- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f3ec8132e00 0x7f3ec81418e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1254000937 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ec81322a0 msgr2=0x7f3ec812e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 --2- 172.21.15.6:0/1254000937 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f3ec81322a0 0x7f3ec812e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.192 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 monclient: found mon.noname-a 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.177+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1254000937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3eb80094f0 con 0x7f3ec8098bb0 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3ec7fff700 1 -- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 msgr2=0x7f3ec809eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3ec7fff700 1 --2- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 0x7f3ec809eac0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8133270 tx=0x7f3eb8016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8006b30 con 0x7f3ec8098bb0 2023-09-30T00:49:40.193 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: dump: 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.194 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3eb8020070 con 0x7f3ec8098bb0 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3ece884700 5 monclient: authenticate success, global_id 4749 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3eb801c6c0 con 0x7f3ec8098bb0 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3eaffff700 10 monclient: dump: 2023-09-30T00:49:40.195 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.196 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 10 monclient: _renew_subs 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.179+0000 7f3ec67fc700 4 set_mon_vals no callback set 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec811e780 con 0x7f3ec8098bb0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 10 monclient: _renew_subs 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec811d5b0 con 0x7f3ec8098bb0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f3ec811d5b0 con 0x7f3ec8098bb0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.180+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f3eb8022030 con 0x7f3ec8098bb0 2023-09-30T00:49:40.197 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3eaffff700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.22:6812/142603 conn(0x7f3eb40357a0 0x7f3eb4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f3eb8049810 con 0x7f3ec8098bb0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3eaffff700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.6:6804/114301 conn(0x7f3eb403b2c0 0x7f3eb403d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 --> v2:172.21.15.6:6804/114301 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3eb403dd40 con 0x7f3eb403b2c0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3eb804bd40 con 0x7f3ec8098bb0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3eaffff700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3ecce21700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.6:6804/114301 conn(0x7f3eb403b2c0 0x7f3eb403d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3ecce21700 10 monclient: get_auth_request con 0x7f3eb403b2c0 auth_method 0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3ec77fe700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.22:6812/142603 conn(0x7f3eb40357a0 0x7f3eb4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.181+0000 7f3ec77fe700 10 monclient: get_auth_request con 0x7f3eb40357a0 auth_method 0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.184+0000 7f3ec77fe700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.22:6812/142603 conn(0x7f3eb40357a0 0x7f3eb4037c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3ec811de30 tx=0x7f3ebc009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.185+0000 7f3ecce21700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.6:6804/114301 conn(0x7f3eb403b2c0 0x7f3eb403d6d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.186+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== osd.0 v2:172.21.15.6:6804/114301 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f3eb403dd40 con 0x7f3eb403b2c0 2023-09-30T00:49:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 --> v2:172.21.15.6:6804/114301 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f3ec8072b00 con 0x7f3eb403b2c0 2023-09-30T00:49:40.200 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3eaffff700 1 -- 172.21.15.6:0/1254000937 <== osd.0 v2:172.21.15.6:6804/114301 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f3ec8072b00 con 0x7f3eb403b2c0 2023-09-30T00:49:40.201 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-09-30T00:49:40.201 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:49:40.201 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-09-30T00:49:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 >> v2:172.21.15.6:6804/114301 conn(0x7f3eb403b2c0 msgr2=0x7f3eb403d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.6:6804/114301 conn(0x7f3eb403b2c0 0x7f3eb403d6d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 >> v2:172.21.15.22:6812/142603 conn(0x7f3eb40357a0 msgr2=0x7f3eb4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 --2- 172.21.15.6:0/1254000937 >> v2:172.21.15.22:6812/142603 conn(0x7f3eb40357a0 0x7f3eb4037c50 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f3ec811de30 tx=0x7f3ebc009040 comp rx=0 tx=0).stop 2023-09-30T00:49:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 10 monclient: shutdown 2023-09-30T00:49:40.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 msgr2=0x7f3ec809eac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 --2- 172.21.15.6:0/1254000937 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f3ec8098bb0 0x7f3ec809eac0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3ec8133270 tx=0x7f3eb8016000 comp rx=0 tx=0).stop 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ecce21700 1 -- 172.21.15.6:0/1254000937 reap_dead start 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.201+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 shutdown_connections 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.202+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 >> 172.21.15.6:0/1254000937 conn(0x7f3ec80a54c0 msgr2=0x7f3ec8131850 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.202+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 shutdown_connections 2023-09-30T00:49:40.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.202+0000 7f3ece884700 1 -- 172.21.15.6:0/1254000937 wait complete. 2023-09-30T00:49:40.214 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2023-09-30T00:49:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.346+0000 7f5b6b386700 10 monclient: get_monmap_and_config 2023-09-30T00:49:40.346 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.346+0000 7f5b6b386700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.346+0000 7f5b6b386700 10 monclient: monmap: 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:40.347060+0000 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:40.347060+0000 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:40.347 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:40.348 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.348 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.347+0000 7f5b6b386700 1 Processor -- start 2023-09-30T00:49:40.349 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.347+0000 7f5b6b386700 1 -- start start 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.347+0000 7f5b6b386700 10 monclient: init 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient: _add_conns ranks=[2,0,4,3,1] 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 0x7f5b64132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): picked mon.noname-b con 0x7f5b64098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5b64098a40 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f5b64098e90 0x7f5b6412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): picked mon.noname-c con 0x7f5b64098e90 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b640a1000 con 0x7f5b64098a40 2023-09-30T00:49:40.350 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5b640a0a50 con 0x7f5b64098e90 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b640a0730 con 0x7f5b64098670 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b6b386700 10 monclient(hunting): authenticate will time out at 1576.645279s 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b68921700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5b64098a40 0x7f5b6412ede0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33468/0 (socket says 172.21.15.6:33468) 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 0x7f5b64132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b68921700 1 -- 172.21.15.6:0/1389907351 learned_addr learned my addr 172.21.15.6:0/1389907351 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): get_auth_request con 0x7f5b64098670 auth_method 0 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.351 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69923700 1 --2- 172.21.15.6:0/1389907351 >> v2:172.21.15.22:3300/0 conn(0x7f5b64098e90 0x7f5b6412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69923700 10 monclient(hunting): get_auth_request con 0x7f5b64098e90 auth_method 0 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69923700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69923700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69923700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient(hunting): handle_auth_done global_id 4796 payload 274 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 1 -- 172.21.15.6:0/1389907351 >> v2:172.21.15.22:3300/0 conn(0x7f5b64098e90 msgr2=0x7f5b6412f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.352 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 1 --2- 172.21.15.6:0/1389907351 >> v2:172.21.15.22:3300/0 conn(0x7f5b64098e90 0x7f5b6412f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 1 -- 172.21.15.6:0/1389907351 >> v1:172.21.15.6:6789/0 conn(0x7f5b64098a40 legacy=0x7f5b6412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 1 monclient: found mon.noname-b 2023-09-30T00:49:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.353 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.348+0000 7f5b69122700 1 -- 172.21.15.6:0/1389907351 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b64068df0 con 0x7f5b64098670 2023-09-30T00:49:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b69122700 1 -- 172.21.15.6:0/1389907351 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 msgr2=0x7f5b64132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b69122700 1 --2- 172.21.15.6:0/1389907351 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 0x7f5b64132d00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5b600098e0 tx=0x7f5b600091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=177435da3f26ba5f server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 1 -- 172.21.15.6:0/1389907351 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b60009e80 con 0x7f5b64098670 2023-09-30T00:49:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: dump: 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.359 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 1 -- 172.21.15.6:0/1389907351 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6000ba30 con 0x7f5b64098670 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 5 monclient: authenticate success, global_id 4796 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 4 set_mon_vals no callback set 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 1 -- 172.21.15.6:0/1389907351 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b6000bdb0 con 0x7f5b64098670 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b5bfff700 10 monclient: dump: 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.361 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 10 monclient: shutdown 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 -- 172.21.15.6:0/1389907351 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 msgr2=0x7f5b64132d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 --2- 172.21.15.6:0/1389907351 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 0x7f5b64132d00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5b600098e0 tx=0x7f5b600091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 -- 172.21.15.6:0/1389907351 shutdown_connections 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 --2- 172.21.15.6:0/1389907351 >> v2:172.21.15.22:3300/0 conn(0x7f5b64098e90 0x7f5b6412f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 --2- 172.21.15.6:0/1389907351 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098670 0x7f5b64132d00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 -- 172.21.15.6:0/1389907351 >> 172.21.15.6:0/1389907351 conn(0x7f5b640a5480 msgr2=0x7f5b640a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.351+0000 7f5b6b386700 1 -- 172.21.15.6:0/1389907351 shutdown_connections 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.352+0000 7f5b6b386700 1 -- 172.21.15.6:0/1389907351 wait complete. 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.352+0000 7f5b6b386700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.352+0000 7f5b6b386700 10 monclient: monmap: 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.352+0000 7f5b6b386700 1 Processor -- start 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.352+0000 7f5b6b386700 1 -- start start 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.352+0000 7f5b6b386700 10 monclient: init 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b64098670 0x7f5b6412d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): picked mon.noname-a con 0x7f5b64098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 0x7f5b6412d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): picked mon.noname-c con 0x7f5b64098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5b64098e90 0x7f5b6409d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): picked mon.noname-b con 0x7f5b64098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b64135390 con 0x7f5b64098670 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6409ce90 con 0x7f5b64098e90 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6412f380 con 0x7f5b64098a40 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b6b386700 10 monclient(hunting): authenticate will time out at 1576.650562s 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b64098670 0x7f5b6412d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 0x7f5b6412d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b64098670 0x7f5b6412d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35766/0 (socket says 172.21.15.6:35766) 2023-09-30T00:49:40.365 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 1 -- 172.21.15.6:0/515327797 learned_addr learned my addr 172.21.15.6:0/515327797 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): get_auth_request con 0x7f5b64098670 auth_method 0 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): get_auth_request con 0x7f5b64098a40 auth_method 0 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69122700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69923700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5b64098e90 0x7f5b6409d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69923700 10 monclient(hunting): get_auth_request con 0x7f5b64098e90 auth_method 0 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69923700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69923700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b69923700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 10 monclient(hunting): handle_auth_done global_id 4799 payload 824 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 1 -- 172.21.15.6:0/515327797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5b64098e90 msgr2=0x7f5b6409d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.353+0000 7f5b68921700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5b64098e90 0x7f5b6409d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 1 -- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b64098670 msgr2=0x7f5b6412d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b64098670 0x7f5b6412d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 1 monclient: found mon.noname-c 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.354+0000 7f5b68921700 1 -- 172.21.15.6:0/515327797 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b60009420 con 0x7f5b64098a40 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b68921700 1 -- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 msgr2=0x7f5b6412d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b68921700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 0x7f5b6412d920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c530 tx=0x7f5b54007f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54005d40 con 0x7f5b64098a40 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: dump: 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.369 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b540070d0 con 0x7f5b64098a40 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 5 monclient: authenticate success, global_id 4799 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b54007490 con 0x7f5b64098a40 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 10 monclient: _renew_subs 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6409cbb0 con 0x7f5b64098a40 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b5b7fe700 4 set_mon_vals no callback set 2023-09-30T00:49:40.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b597fa700 10 monclient: dump: 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.371 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 10 monclient: _renew_subs 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.356+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6412de60 con 0x7f5b64098a40 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.357+0000 7f5b6b386700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.357+0000 7f5b6b386700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.357+0000 7f5b6b386700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.357+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6409d020 con 0x7f5b64098a40 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.357+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5b5401b3b0 con 0x7f5b64098a40 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.358+0000 7f5b597fa700 1 --2- 172.21.15.6:0/515327797 >> v2:172.21.15.22:6812/142603 conn(0x7f5b480357a0 0x7f5b48037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.358+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5b54047230 con 0x7f5b64098a40 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.358+0000 7f5b69122700 1 --2- 172.21.15.6:0/515327797 >> v2:172.21.15.22:6812/142603 conn(0x7f5b480357a0 0x7f5b48037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.358+0000 7f5b69122700 10 monclient: get_auth_request con 0x7f5b480357a0 auth_method 0 2023-09-30T00:49:40.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.361+0000 7f5b69122700 1 --2- 172.21.15.6:0/515327797 >> v2:172.21.15.22:6812/142603 conn(0x7f5b480357a0 0x7f5b48037c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5b600030c0 tx=0x7f5b60016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.362+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f5b5400f490 con 0x7f5b64098a40 2023-09-30T00:49:40.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.362+0000 7f5b597fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:40.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.362+0000 7f5b597fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:40.564 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.565+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b6412e050 con 0x7f5b480357a0 2023-09-30T00:49:40.565 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.565+0000 7f5b597fa700 1 -- 172.21.15.6:0/515327797 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f5b6412e050 con 0x7f5b480357a0 2023-09-30T00:49:40.567 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 >> v2:172.21.15.22:6812/142603 conn(0x7f5b480357a0 msgr2=0x7f5b48037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 --2- 172.21.15.6:0/515327797 >> v2:172.21.15.22:6812/142603 conn(0x7f5b480357a0 0x7f5b48037c50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5b600030c0 tx=0x7f5b60016000 comp rx=0 tx=0).stop 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 10 monclient: shutdown 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 msgr2=0x7f5b6412d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 0x7f5b6412d920 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5b5400c530 tx=0x7f5b54007f90 comp rx=0 tx=0).stop 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 shutdown_connections 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 --2- 172.21.15.6:0/515327797 >> v2:172.21.15.22:6812/142603 conn(0x7f5b480357a0 0x7f5b48037c50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5b64098e90 0x7f5b6409d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5b64098670 0x7f5b6412d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.568 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 --2- 172.21.15.6:0/515327797 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5b64098a40 0x7f5b6412d920 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 >> 172.21.15.6:0/515327797 conn(0x7f5b640a5480 msgr2=0x7f5b64069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 shutdown_connections 2023-09-30T00:49:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.568+0000 7f5b6b386700 1 -- 172.21.15.6:0/515327797 wait complete. 2023-09-30T00:49:40.569 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 0 to deep-scrub 2023-09-30T00:49:40.580 INFO:tasks.ceph:Scrubbing osd.1 2023-09-30T00:49:40.581 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:49:40.715 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.716+0000 7f94c9b8c700 10 monclient: get_monmap_and_config 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.716+0000 7f94c9b8c700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.716+0000 7f94c9b8c700 10 monclient: monmap: 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:40.716473+0000 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:40.716473+0000 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.716 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:40.717 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:40.717 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.720 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 Processor -- start 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 -- start start 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient: init 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient: _add_conns ranks=[0,2,4,1,3] 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f94c4098530 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 0x7f94c4132730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): picked mon.noname-b con 0x7f94c4132360 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f94c41374a0 0x7f94c4139880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): picked mon.noname-c con 0x7f94c41374a0 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94c4096af0 con 0x7f94c4098530 2023-09-30T00:49:40.721 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f94c406a420 con 0x7f94c41374a0 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c4097000 con 0x7f94c4132360 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c9b8c700 10 monclient(hunting): authenticate will time out at 1577.014698s 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 0x7f94c4132730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c37fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f94c4098530 0x7f94c4098900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33470/0 (socket says 172.21.15.6:33470) 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c37fe700 1 -- 172.21.15.6:0/2450423800 learned_addr learned my addr 172.21.15.6:0/2450423800 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 10 monclient(hunting): get_auth_request con 0x7f94c4132360 auth_method 0 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.722 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c3fff700 1 --2- 172.21.15.6:0/2450423800 >> v2:172.21.15.22:3300/0 conn(0x7f94c41374a0 0x7f94c4139880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.717+0000 7f94c2ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): get_auth_request con 0x7f94c41374a0 auth_method 0 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.723 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 10 monclient(hunting): handle_auth_done global_id 4802 payload 274 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 1 -- 172.21.15.6:0/2450423800 >> v2:172.21.15.22:3300/0 conn(0x7f94c41374a0 msgr2=0x7f94c4139880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 1 --2- 172.21.15.6:0/2450423800 >> v2:172.21.15.22:3300/0 conn(0x7f94c41374a0 0x7f94c4139880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 1 -- 172.21.15.6:0/2450423800 >> v1:172.21.15.6:6789/0 conn(0x7f94c4098530 legacy=0x7f94c4098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 1 monclient: found mon.noname-b 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.718+0000 7f94c2ffd700 1 -- 172.21.15.6:0/2450423800 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94c4132c70 con 0x7f94c4132360 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c2ffd700 1 -- 172.21.15.6:0/2450423800 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 msgr2=0x7f94c4132730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c2ffd700 1 --2- 172.21.15.6:0/2450423800 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 0x7f94c4132730 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f94b80098e0 tx=0x7f94b80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=534d638d2e5149cd server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 1 -- 172.21.15.6:0/2450423800 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8011840 con 0x7f94c4132360 2023-09-30T00:49:40.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: dump: 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.725 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 1 -- 172.21.15.6:0/2450423800 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b80119a0 con 0x7f94c4132360 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c9b8c700 5 monclient: authenticate success, global_id 4802 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c9b8c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c9b8c700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c9b8c700 4 set_mon_vals no callback set 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 1 -- 172.21.15.6:0/2450423800 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b8011d40 con 0x7f94c4132360 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.720+0000 7f94c27fc700 10 monclient: dump: 2023-09-30T00:49:40.726 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 10 monclient: shutdown 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.727 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 -- 172.21.15.6:0/2450423800 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 msgr2=0x7f94c4132730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 --2- 172.21.15.6:0/2450423800 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 0x7f94c4132730 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f94b80098e0 tx=0x7f94b80091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 -- 172.21.15.6:0/2450423800 shutdown_connections 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 --2- 172.21.15.6:0/2450423800 >> v2:172.21.15.22:3300/0 conn(0x7f94c41374a0 0x7f94c4139880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 --2- 172.21.15.6:0/2450423800 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4132360 0x7f94c4132730 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 -- 172.21.15.6:0/2450423800 >> 172.21.15.6:0/2450423800 conn(0x7f94c40a5460 msgr2=0x7f94c40a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 -- 172.21.15.6:0/2450423800 shutdown_connections 2023-09-30T00:49:40.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.721+0000 7f94c9b8c700 1 -- 172.21.15.6:0/2450423800 wait complete. 2023-09-30T00:49:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.722+0000 7f94c9b8c700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.722+0000 7f94c9b8c700 10 monclient: monmap: 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:40.734 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.722+0000 7f94c9b8c700 1 Processor -- start 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.722+0000 7f94c9b8c700 1 -- start start 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.722+0000 7f94c9b8c700 10 monclient: init 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 0x7f94c4141a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): picked mon.noname-c con 0x7f94c4098530 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94c4132360 0x7f94c4141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): picked mon.noname-b con 0x7f94c4132360 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c41374a0 0x7f94c4142500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): picked mon.noname-a con 0x7f94c41374a0 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41359c0 con 0x7f94c41374a0 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94c41348f0 con 0x7f94c4132360 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f94c40a3960 con 0x7f94c4098530 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c9b8c700 10 monclient(hunting): authenticate will time out at 1577.020554s 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 0x7f94c4141a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 0x7f94c4141a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40116/0 (socket says 172.21.15.6:40116) 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 1 -- 172.21.15.6:0/767348398 learned_addr learned my addr 172.21.15.6:0/767348398 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): get_auth_request con 0x7f94c4098530 auth_method 0 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.736 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c41374a0 0x7f94c4142500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): get_auth_request con 0x7f94c41374a0 auth_method 0 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.737 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c2ffd700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94c4132360 0x7f94c4141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c3fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c2ffd700 10 monclient(hunting): get_auth_request con 0x7f94c4132360 auth_method 0 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c2ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c2ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c2ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 10 monclient(hunting): handle_auth_done global_id 4805 payload 824 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.723+0000 7f94c37fe700 1 -- 172.21.15.6:0/767348398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94c4132360 msgr2=0x7f94c4141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94c4132360 0x7f94c4141fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 1 -- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c41374a0 msgr2=0x7f94c4142500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c41374a0 0x7f94c4142500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.738 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 1 monclient: found mon.noname-c 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c37fe700 1 -- 172.21.15.6:0/767348398 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94b8009420 con 0x7f94c4098530 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c2ffd700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f94c4132360 0x7f94c4141fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.724+0000 7f94c3fff700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f94c41374a0 0x7f94c4142500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c37fe700 1 -- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 msgr2=0x7f94c4141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c37fe700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 0x7f94c4141a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f94b400c5e0 tx=0x7f94b4004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4005d40 con 0x7f94c4098530 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: dump: 2023-09-30T00:49:40.739 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 5 monclient: authenticate success, global_id 4805 2023-09-30T00:49:40.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94b40070d0 con 0x7f94c4098530 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 10 monclient: _renew_subs 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94b4007490 con 0x7f94c4098530 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94c41356e0 con 0x7f94c4098530 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94ab7fe700 10 monclient: dump: 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.741 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c1ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 10 monclient: _renew_subs 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.726+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94c409ea60 con 0x7f94c4098530 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.727+0000 7f94c9b8c700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.727+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f94c4097f50 con 0x7f94c4098530 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.727+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f94b401b3b0 con 0x7f94c4098530 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94ab7fe700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.22:6812/142603 conn(0x7f94b00357f0 0x7f94b0037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f94b4046f50 con 0x7f94c4098530 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94ab7fe700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.6:6800/114300 conn(0x7f94b003b390 0x7f94b003d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 --> v2:172.21.15.6:6800/114300 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f94b003de10 con 0x7f94b003b390 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f94b40494e0 con 0x7f94c4098530 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94ab7fe700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94c3fff700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.6:6800/114300 conn(0x7f94b003b390 0x7f94b003d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94c2ffd700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.22:6812/142603 conn(0x7f94b00357f0 0x7f94b0037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94c3fff700 10 monclient: get_auth_request con 0x7f94b003b390 auth_method 0 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.728+0000 7f94c2ffd700 10 monclient: get_auth_request con 0x7f94b00357f0 auth_method 0 2023-09-30T00:49:40.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.730+0000 7f94c2ffd700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.22:6812/142603 conn(0x7f94b00357f0 0x7f94b0037ca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f94b80030c0 tx=0x7f94b8016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.732+0000 7f94c3fff700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.6:6800/114300 conn(0x7f94b003b390 0x7f94b003d7a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.733+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== osd.1 v2:172.21.15.6:6800/114300 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f94b003de10 con 0x7f94b003b390 2023-09-30T00:49:40.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.747+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 --> v2:172.21.15.6:6800/114300 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f94c4072b00 con 0x7f94b003b390 2023-09-30T00:49:40.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94ab7fe700 1 -- 172.21.15.6:0/767348398 <== osd.1 v2:172.21.15.6:6800/114300 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f94c4072b00 con 0x7f94b003b390 2023-09-30T00:49:40.747 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-09-30T00:49:40.748 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:49:40.748 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 >> v2:172.21.15.6:6800/114300 conn(0x7f94b003b390 msgr2=0x7f94b003d7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.6:6800/114300 conn(0x7f94b003b390 0x7f94b003d7a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 >> v2:172.21.15.22:6812/142603 conn(0x7f94b00357f0 msgr2=0x7f94b0037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 --2- 172.21.15.6:0/767348398 >> v2:172.21.15.22:6812/142603 conn(0x7f94b00357f0 0x7f94b0037ca0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f94b80030c0 tx=0x7f94b8016000 comp rx=0 tx=0).stop 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 10 monclient: shutdown 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 msgr2=0x7f94c4141a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 --2- 172.21.15.6:0/767348398 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f94c4098530 0x7f94c4141a80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f94b400c5e0 tx=0x7f94b4004a80 comp rx=0 tx=0).stop 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c3fff700 1 -- 172.21.15.6:0/767348398 reap_dead start 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 shutdown_connections 2023-09-30T00:49:40.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 >> 172.21.15.6:0/767348398 conn(0x7f94c40a5460 msgr2=0x7f94c412ef00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 shutdown_connections 2023-09-30T00:49:40.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.748+0000 7f94c9b8c700 1 -- 172.21.15.6:0/767348398 wait complete. 2023-09-30T00:49:40.760 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2023-09-30T00:49:40.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.892+0000 7f4fed282700 10 monclient: get_monmap_and_config 2023-09-30T00:49:40.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.892+0000 7f4fed282700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.892 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.892+0000 7f4fed282700 10 monclient: monmap: 2023-09-30T00:49:40.892 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.892 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.892 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:40.892918+0000 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:40.892918+0000 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:40.893 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 Processor -- start 2023-09-30T00:49:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 -- start start 2023-09-30T00:49:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient: init 2023-09-30T00:49:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.897 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient: _add_conns ranks=[3,4,1,0,2] 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4fe8098600 0x7f4fe80989d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): picked mon.noname-a con 0x7f4fe8098600 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f4fe8132410 0x7f4fe8132820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): picked mon.noname-c con 0x7f4fe8132410 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f4fe8068df0 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4fe80a0f60 con 0x7f4fe8068df0 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe806a280 con 0x7f4fe8098600 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4fe8132d60 con 0x7f4fe8132410 2023-09-30T00:49:40.898 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.893+0000 7f4fed282700 10 monclient(hunting): authenticate will time out at 1577.190935s 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4fe8098600 0x7f4fe80989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f4fe8098600 0x7f4fe80989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35790/0 (socket says 172.21.15.6:35790) 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/1657251670 learned_addr learned my addr 172.21.15.6:0/1657251670 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): get_auth_request con 0x7f4fe8098600 auth_method 0 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.899 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe67fc700 1 --2- 172.21.15.6:0/1657251670 >> v2:172.21.15.22:3300/0 conn(0x7f4fe8132410 0x7f4fe8132820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe67fc700 10 monclient(hunting): get_auth_request con 0x7f4fe8132410 auth_method 0 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_done global_id 4758 payload 274 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/1657251670 >> v2:172.21.15.22:3300/0 conn(0x7f4fe8132410 msgr2=0x7f4fe8132820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.900 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 --2- 172.21.15.6:0/1657251670 >> v2:172.21.15.22:3300/0 conn(0x7f4fe8132410 0x7f4fe8132820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/1657251670 >> v1:172.21.15.22:6789/0 conn(0x7f4fe8068df0 legacy=0x7f4fe8069270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 monclient: found mon.noname-a 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.894+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/1657251670 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fe80a1cb0 con 0x7f4fe8098600 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/1657251670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8098600 msgr2=0x7f4fe80989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe6ffd700 1 --2- 172.21.15.6:0/1657251670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8098600 0x7f4fe80989d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80a65d0 tx=0x7f4fdc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a9ad8b48da8b745 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 1 -- 172.21.15.6:0/1657251670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc00ba30 con 0x7f4fe8098600 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: dump: 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.902 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fed282700 5 monclient: authenticate success, global_id 4758 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 1 -- 172.21.15.6:0/1657251670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc00bb90 con 0x7f4fe8098600 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fed282700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fed282700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fed282700 4 set_mon_vals no callback set 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 1 -- 172.21.15.6:0/1657251670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc011420 con 0x7f4fe8098600 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.896+0000 7f4fe5ffb700 10 monclient: dump: 2023-09-30T00:49:40.903 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.904 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 10 monclient: shutdown 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 -- 172.21.15.6:0/1657251670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8098600 msgr2=0x7f4fe80989d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 --2- 172.21.15.6:0/1657251670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8098600 0x7f4fe80989d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f4fe80a65d0 tx=0x7f4fdc009250 comp rx=0 tx=0).stop 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 -- 172.21.15.6:0/1657251670 shutdown_connections 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 --2- 172.21.15.6:0/1657251670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8098600 0x7f4fe80989d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 --2- 172.21.15.6:0/1657251670 >> v2:172.21.15.22:3300/0 conn(0x7f4fe8132410 0x7f4fe8132820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 -- 172.21.15.6:0/1657251670 >> 172.21.15.6:0/1657251670 conn(0x7f4fe80a54c0 msgr2=0x7f4fe80a58c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 -- 172.21.15.6:0/1657251670 shutdown_connections 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.897+0000 7f4fed282700 1 -- 172.21.15.6:0/1657251670 wait complete. 2023-09-30T00:49:40.905 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.898+0000 7f4fed282700 10 monclient: build_initial_monmap 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.898+0000 7f4fed282700 10 monclient: monmap: 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:40.906 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.898+0000 7f4fed282700 1 Processor -- start 2023-09-30T00:49:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.898+0000 7f4fed282700 1 -- start start 2023-09-30T00:49:40.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.898+0000 7f4fed282700 10 monclient: init 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.898+0000 7f4fed282700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 0x7f4fe811c180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): picked mon.noname-c con 0x7f4fe8068df0 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4fe8098600 0x7f4fe811c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): picked mon.noname-b con 0x7f4fe8098600 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 0x7f4fe811cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): picked mon.noname-a con 0x7f4fe8132410 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe80a1460 con 0x7f4fe8132410 2023-09-30T00:49:40.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe80a3b80 con 0x7f4fe8098600 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4fe811fbc0 con 0x7f4fe8068df0 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fed282700 10 monclient(hunting): authenticate will time out at 1577.196080s 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 0x7f4fe811cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 0x7f4fe811c180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 0x7f4fe811c180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40132/0 (socket says 172.21.15.6:40132) 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 0x7f4fe811cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35806/0 (socket says 172.21.15.6:35806) 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/3296333615 learned_addr learned my addr 172.21.15.6:0/3296333615 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:40.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): get_auth_request con 0x7f4fe8068df0 auth_method 0 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): get_auth_request con 0x7f4fe8132410 auth_method 0 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:40.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe67fc700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4fe8098600 0x7f4fe811c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe67fc700 10 monclient(hunting): get_auth_request con 0x7f4fe8098600 auth_method 0 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe67fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe67fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe67fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient(hunting): handle_auth_done global_id 4808 payload 824 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/3296333615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4fe8098600 msgr2=0x7f4fe811c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4fe8098600 0x7f4fe811c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 msgr2=0x7f4fe811cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 0x7f4fe811cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 monclient: found mon.noname-c 2023-09-30T00:49:40.915 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/3296333615 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fdc0094f0 con 0x7f4fe8068df0 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.899+0000 7f4fe77fe700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 0x7f4fe811cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.901+0000 7f4fe6ffd700 1 -- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 msgr2=0x7f4fe811c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.901+0000 7f4fe6ffd700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 0x7f4fe811c180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4fe806a630 tx=0x7f4fdc016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc01b470 con 0x7f4fe8068df0 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: dump: 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.916 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4fdc01b5d0 con 0x7f4fe8068df0 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 5 monclient: authenticate success, global_id 4808 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:40.917 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 10 monclient: _renew_subs 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4fdc01b990 con 0x7f4fe8068df0 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fe811f850 con 0x7f4fe8068df0 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fe57fa700 4 set_mon_vals no callback set 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fceffd700 10 monclient: dump: 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.918 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 10 monclient: _renew_subs 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.902+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fe811e960 con 0x7f4fe8068df0 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f4fdc01c450 con 0x7f4fe8068df0 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fed282700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fed282700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:40.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fed282700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fe811ff50 con 0x7f4fe8068df0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fceffd700 1 --2- 172.21.15.6:0/3296333615 >> v2:172.21.15.22:6812/142603 conn(0x7f4fd40357f0 0x7f4fd4037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f4fdc049500 con 0x7f4fe8068df0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fe67fc700 1 --2- 172.21.15.6:0/3296333615 >> v2:172.21.15.22:6812/142603 conn(0x7f4fd40357f0 0x7f4fd4037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.903+0000 7f4fe67fc700 10 monclient: get_auth_request con 0x7f4fd40357f0 auth_method 0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.906+0000 7f4fe67fc700 1 --2- 172.21.15.6:0/3296333615 >> v2:172.21.15.22:6812/142603 conn(0x7f4fd40357f0 0x7f4fd4037ca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4fe811c0a0 tx=0x7f4fd8008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.908+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f4fe811ff50 con 0x7f4fe8068df0 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.908+0000 7f4fceffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:40.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:40.908+0000 7f4fceffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:41.107 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.108+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f4fe811fd00 con 0x7f4fd40357f0 2023-09-30T00:49:41.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.108+0000 7f4fceffd700 1 -- 172.21.15.6:0/3296333615 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4fd800f620 con 0x7f4fd40357f0 2023-09-30T00:49:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 >> v2:172.21.15.22:6812/142603 conn(0x7f4fd40357f0 msgr2=0x7f4fd4037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 --2- 172.21.15.6:0/3296333615 >> v2:172.21.15.22:6812/142603 conn(0x7f4fd40357f0 0x7f4fd4037ca0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f4fe811c0a0 tx=0x7f4fd8008040 comp rx=0 tx=0).stop 2023-09-30T00:49:41.110 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 10 monclient: shutdown 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 msgr2=0x7f4fe811c180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 0x7f4fe811c180 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f4fe806a630 tx=0x7f4fdc016000 comp rx=0 tx=0).stop 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 shutdown_connections 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 --2- 172.21.15.6:0/3296333615 >> v2:172.21.15.22:6812/142603 conn(0x7f4fd40357f0 0x7f4fd4037ca0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f4fe8132410 0x7f4fe811cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f4fe8068df0 0x7f4fe811c180 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 --2- 172.21.15.6:0/3296333615 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f4fe8098600 0x7f4fe811c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.110+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 >> 172.21.15.6:0/3296333615 conn(0x7f4fe80a54c0 msgr2=0x7f4fe8130300 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.111+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 shutdown_connections 2023-09-30T00:49:41.111 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.111+0000 7f4fed282700 1 -- 172.21.15.6:0/3296333615 wait complete. 2023-09-30T00:49:41.112 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 1 to deep-scrub 2023-09-30T00:49:41.123 INFO:tasks.ceph:Scrubbing osd.2 2023-09-30T00:49:41.123 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:49:41.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.258+0000 7f01185a0700 10 monclient: get_monmap_and_config 2023-09-30T00:49:41.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.258+0000 7f01185a0700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.258 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.258+0000 7f01185a0700 10 monclient: monmap: 2023-09-30T00:49:41.258 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.258 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:41.258603+0000 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:41.258603+0000 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.258+0000 7f01185a0700 1 Processor -- start 2023-09-30T00:49:41.259 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 1 -- start start 2023-09-30T00:49:41.260 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient: init 2023-09-30T00:49:41.266 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient: _add_conns ranks=[4,1,3,0,2] 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f0110098530 0x7f0110098900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient(hunting): picked mon.noname-c con 0x7f0110098530 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f0110098e40 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0110132a70 0x7f01101394b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient(hunting): picked mon.noname-a con 0x7f0110132a70 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0110096af0 con 0x7f0110098e40 2023-09-30T00:49:41.267 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.259+0000 7f01185a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f01185a0700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f011006a420 con 0x7f0110132a70 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f01185a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f01185a0700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0110097000 con 0x7f0110098530 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f01185a0700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f01185a0700 10 monclient(hunting): authenticate will time out at 1577.557027s 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0110132a70 0x7f01101394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0110132a70 0x7f01101394b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35822/0 (socket says 172.21.15.6:35822) 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 -- 172.21.15.6:0/608484670 learned_addr learned my addr 172.21.15.6:0/608484670 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): get_auth_request con 0x7f0110132a70 auth_method 0 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.268 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f011633c700 1 --2- 172.21.15.6:0/608484670 >> v2:172.21.15.22:3300/0 conn(0x7f0110098530 0x7f0110098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f011633c700 10 monclient(hunting): get_auth_request con 0x7f0110098530 auth_method 0 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f011633c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f011633c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.269 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f011633c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient(hunting): handle_auth_done global_id 4764 payload 274 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 -- 172.21.15.6:0/608484670 >> v2:172.21.15.22:3300/0 conn(0x7f0110098530 msgr2=0x7f0110098900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 --2- 172.21.15.6:0/608484670 >> v2:172.21.15.22:3300/0 conn(0x7f0110098530 0x7f0110098900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 -- 172.21.15.6:0/608484670 >> v1:172.21.15.22:6789/0 conn(0x7f0110098e40 legacy=0x7f0110132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 monclient: found mon.noname-a 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.260+0000 7f0116b3d700 1 -- 172.21.15.6:0/608484670 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0110096d20 con 0x7f0110132a70 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f0116b3d700 1 -- 172.21.15.6:0/608484670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110132a70 msgr2=0x7f01101394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f0116b3d700 1 --2- 172.21.15.6:0/608484670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110132a70 0x7f01101394b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f011006a670 tx=0x7f0108009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40fcaad094735b07 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.270 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 1 -- 172.21.15.6:0/608484670 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f010800e070 con 0x7f0110132a70 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: dump: 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.271 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 1 -- 172.21.15.6:0/608484670 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f010800bc20 con 0x7f0110132a70 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f01185a0700 5 monclient: authenticate success, global_id 4764 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f01185a0700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f01185a0700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 1 -- 172.21.15.6:0/608484670 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01080092e0 con 0x7f0110132a70 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f01185a0700 4 set_mon_vals no callback set 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:41.272 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.262+0000 7f011533a700 10 monclient: dump: 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 10 monclient: shutdown 2023-09-30T00:49:41.273 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 -- 172.21.15.6:0/608484670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110132a70 msgr2=0x7f01101394b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 --2- 172.21.15.6:0/608484670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110132a70 0x7f01101394b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f011006a670 tx=0x7f0108009840 comp rx=0 tx=0).stop 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 -- 172.21.15.6:0/608484670 shutdown_connections 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 --2- 172.21.15.6:0/608484670 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110132a70 0x7f01101394b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 --2- 172.21.15.6:0/608484670 >> v2:172.21.15.22:3300/0 conn(0x7f0110098530 0x7f0110098900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 -- 172.21.15.6:0/608484670 >> 172.21.15.6:0/608484670 conn(0x7f01100a5460 msgr2=0x7f01100a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 -- 172.21.15.6:0/608484670 shutdown_connections 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.263+0000 7f01185a0700 1 -- 172.21.15.6:0/608484670 wait complete. 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.264+0000 7f01185a0700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.264+0000 7f01185a0700 10 monclient: monmap: 2023-09-30T00:49:41.274 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.264+0000 7f01185a0700 1 Processor -- start 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.264+0000 7f01185a0700 1 -- start start 2023-09-30T00:49:41.275 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.264+0000 7f01185a0700 10 monclient: init 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0110098530 0x7f0110138590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): picked mon.noname-b con 0x7f0110098530 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 0x7f0110134250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): picked mon.noname-a con 0x7f0110098e40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0110132a70 0x7f0110134790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): picked mon.noname-c con 0x7f0110132a70 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110139600 con 0x7f0110098e40 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.276 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0110137c40 con 0x7f0110098530 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01101356c0 con 0x7f0110132a70 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f01185a0700 10 monclient(hunting): authenticate will time out at 1577.562221s 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 0x7f0110134250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0110132a70 0x7f0110134790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 0x7f0110134250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35830/0 (socket says 172.21.15.6:35830) 2023-09-30T00:49:41.277 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0110132a70 0x7f0110134790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40146/0 (socket says 172.21.15.6:40146) 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 -- 172.21.15.6:0/795606504 learned_addr learned my addr 172.21.15.6:0/795606504 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): get_auth_request con 0x7f0110098e40 auth_method 0 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): get_auth_request con 0x7f0110132a70 auth_method 0 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.278 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0116b3d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 1 --2- 172.21.15.6:0/795606504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0110098530 0x7f0110138590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): get_auth_request con 0x7f0110098530 auth_method 0 2023-09-30T00:49:41.279 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f011633c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient(hunting): handle_auth_done global_id 4767 payload 824 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 -- 172.21.15.6:0/795606504 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0110132a70 msgr2=0x7f0110134790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 --2- 172.21.15.6:0/795606504 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0110132a70 0x7f0110134790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 -- 172.21.15.6:0/795606504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0110098530 msgr2=0x7f0110138590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 --2- 172.21.15.6:0/795606504 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0110098530 0x7f0110138590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.280 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 monclient: found mon.noname-a 2023-09-30T00:49:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:41.281 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.265+0000 7f0115b3b700 1 -- 172.21.15.6:0/795606504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0108009ae0 con 0x7f0110098e40 2023-09-30T00:49:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.267+0000 7f0115b3b700 1 -- 172.21.15.6:0/795606504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 msgr2=0x7f0110134250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.267+0000 7f0115b3b700 1 --2- 172.21.15.6:0/795606504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 0x7f0110134250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f010000c5d0 tx=0x7f0100007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.283 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f01000117a0 con 0x7f0110098e40 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: dump: 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.284 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100011900 con 0x7f0110098e40 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 5 monclient: authenticate success, global_id 4767 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 10 monclient: _renew_subs 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01101378d0 con 0x7f0110098e40 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0100011cc0 con 0x7f0110098e40 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.285 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01067fc700 10 monclient: dump: 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.286 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f0114b39700 4 set_mon_vals no callback set 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 10 monclient: _renew_subs 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0110138ad0 con 0x7f0110098e40 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.268+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0110072b00 con 0x7f0110098e40 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0100012630 con 0x7f0110098e40 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.22:6812/142603 conn(0x7f00f40357f0 0x7f00f4037ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f01000475a0 con 0x7f0110098e40 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.6:6808/114302 conn(0x7f00f403b390 0x7f00f403d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 --> v2:172.21.15.6:6808/114302 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f00f403de10 con 0x7f00f403b390 2023-09-30T00:49:41.287 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0100049ad0 con 0x7f0110098e40 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f01067fc700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f0116b3d700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.6:6808/114302 conn(0x7f00f403b390 0x7f00f403d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f011633c700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.22:6812/142603 conn(0x7f00f40357f0 0x7f00f4037ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f0116b3d700 10 monclient: get_auth_request con 0x7f00f403b390 auth_method 0 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.269+0000 7f011633c700 10 monclient: get_auth_request con 0x7f00f40357f0 auth_method 0 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.272+0000 7f011633c700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.22:6812/142603 conn(0x7f00f40357f0 0x7f00f4037ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f010c006fd0 tx=0x7f010c008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.274+0000 7f0116b3d700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.6:6808/114302 conn(0x7f00f403b390 0x7f00f403d7a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.288 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.274+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== osd.2 v2:172.21.15.6:6808/114302 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f00f403de10 con 0x7f00f403b390 2023-09-30T00:49:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 --> v2:172.21.15.6:6808/114302 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f0110136a40 con 0x7f00f403b390 2023-09-30T00:49:41.289 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01067fc700 1 -- 172.21.15.6:0/795606504 <== osd.2 v2:172.21.15.6:6808/114302 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f0110136a40 con 0x7f00f403b390 2023-09-30T00:49:41.289 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-09-30T00:49:41.289 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:49:41.289 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-09-30T00:49:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 >> v2:172.21.15.6:6808/114302 conn(0x7f00f403b390 msgr2=0x7f00f403d7a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.6:6808/114302 conn(0x7f00f403b390 0x7f00f403d7a0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.290 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 >> v2:172.21.15.22:6812/142603 conn(0x7f00f40357f0 msgr2=0x7f00f4037ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 --2- 172.21.15.6:0/795606504 >> v2:172.21.15.22:6812/142603 conn(0x7f00f40357f0 0x7f00f4037ca0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f010c006fd0 tx=0x7f010c008040 comp rx=0 tx=0).stop 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 10 monclient: shutdown 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 msgr2=0x7f0110134250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.289+0000 7f01185a0700 1 --2- 172.21.15.6:0/795606504 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0110098e40 0x7f0110134250 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f010000c5d0 tx=0x7f0100007f60 comp rx=0 tx=0).stop 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.290+0000 7f0116b3d700 1 -- 172.21.15.6:0/795606504 reap_dead start 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.290+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 shutdown_connections 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.290+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 >> 172.21.15.6:0/795606504 conn(0x7f01100a5460 msgr2=0x7f0110068dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.290+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 shutdown_connections 2023-09-30T00:49:41.291 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.290+0000 7f01185a0700 1 -- 172.21.15.6:0/795606504 wait complete. 2023-09-30T00:49:41.302 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2023-09-30T00:49:41.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.433+0000 7fd76bef4700 10 monclient: get_monmap_and_config 2023-09-30T00:49:41.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.433+0000 7fd76bef4700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.433 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.433+0000 7fd76bef4700 10 monclient: monmap: 2023-09-30T00:49:41.433 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:41.433929+0000 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:41.433929+0000 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:41.434 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.435 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.434+0000 7fd76bef4700 1 Processor -- start 2023-09-30T00:49:41.436 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.434+0000 7fd76bef4700 1 -- start start 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.434+0000 7fd76bef4700 10 monclient: init 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.434+0000 7fd76bef4700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient: _add_conns ranks=[1,0,2,4,3] 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fd764098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fd764132d00 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 0x7fd76413b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient(hunting): picked mon.noname-b con 0x7fd764138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640a1000 con 0x7fd764132d00 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7640a0730 con 0x7fd764098670 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd764099150 con 0x7fd764138d40 2023-09-30T00:49:41.437 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76bef4700 10 monclient(hunting): authenticate will time out at 1577.732105s 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 0x7fd76413b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76948f700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fd764132d00 0x7fd764137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33472/0 (socket says 172.21.15.6:33472) 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76948f700 1 -- 172.21.15.6:0/3201651196 learned_addr learned my addr 172.21.15.6:0/3201651196 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): get_auth_request con 0x7fd764138d40 auth_method 0 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.438 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient(hunting): handle_auth_done global_id 4814 payload 274 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 1 -- 172.21.15.6:0/3201651196 >> v1:172.21.15.22:6789/0 conn(0x7fd764098670 legacy=0x7fd764098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 1 -- 172.21.15.6:0/3201651196 >> v1:172.21.15.6:6789/0 conn(0x7fd764132d00 legacy=0x7fd764137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 1 monclient: found mon.noname-b 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.439 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.435+0000 7fd76a491700 1 -- 172.21.15.6:0/3201651196 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd764068df0 con 0x7fd764138d40 2023-09-30T00:49:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.437+0000 7fd76a491700 1 -- 172.21.15.6:0/3201651196 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 msgr2=0x7fd76413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.437+0000 7fd76a491700 1 --2- 172.21.15.6:0/3201651196 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 0x7fd76413b120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009e90 tx=0x7fd75c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=90fdde22d2107f63 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.444 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 1 -- 172.21.15.6:0/3201651196 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00e070 con 0x7fd764138d40 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: dump: 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.445 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 1 -- 172.21.15.6:0/3201651196 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd75c012430 con 0x7fd764138d40 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 5 monclient: authenticate success, global_id 4814 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 1 -- 172.21.15.6:0/3201651196 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd75c00bea0 con 0x7fd764138d40 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 4 set_mon_vals no callback set 2023-09-30T00:49:41.446 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd768c8e700 10 monclient: dump: 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.447 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 10 monclient: shutdown 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 -- 172.21.15.6:0/3201651196 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 msgr2=0x7fd76413b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 --2- 172.21.15.6:0/3201651196 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 0x7fd76413b120 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd75c009e90 tx=0x7fd75c009780 comp rx=0 tx=0).stop 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 -- 172.21.15.6:0/3201651196 shutdown_connections 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 --2- 172.21.15.6:0/3201651196 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764138d40 0x7fd76413b120 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 -- 172.21.15.6:0/3201651196 >> 172.21.15.6:0/3201651196 conn(0x7fd7640a5480 msgr2=0x7fd7640a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 -- 172.21.15.6:0/3201651196 shutdown_connections 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.438+0000 7fd76bef4700 1 -- 172.21.15.6:0/3201651196 wait complete. 2023-09-30T00:49:41.448 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.439+0000 7fd76bef4700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.439+0000 7fd76bef4700 10 monclient: monmap: 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:41.449 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.439+0000 7fd76bef4700 1 Processor -- start 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.439+0000 7fd76bef4700 1 -- start start 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.439+0000 7fd76bef4700 10 monclient: init 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 0x7fd764141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): picked mon.noname-c con 0x7fd764098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd764132d00 0x7fd7641383d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): picked mon.noname-a con 0x7fd764132d00 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:41.450 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd764138d40 0x7fd76409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): picked mon.noname-b con 0x7fd764138d40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641302b0 con 0x7fd764132d00 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd76409ed80 con 0x7fd764138d40 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7641424c0 con 0x7fd764098670 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76bef4700 10 monclient(hunting): authenticate will time out at 1577.737221s 2023-09-30T00:49:41.451 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 0x7fd764141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 0x7fd764141670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40168/0 (socket says 172.21.15.6:40168) 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 -- 172.21.15.6:0/4195410516 learned_addr learned my addr 172.21.15.6:0/4195410516 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd764132d00 0x7fd7641383d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): get_auth_request con 0x7fd764098670 auth_method 0 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): get_auth_request con 0x7fd764132d00 auth_method 0 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.452 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76948f700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient(hunting): handle_auth_done global_id 4817 payload 824 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76a491700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd764138d40 0x7fd76409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 -- 172.21.15.6:0/4195410516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd764138d40 msgr2=0x7fd76409c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd764138d40 0x7fd76409c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 -- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd764132d00 msgr2=0x7fd7641383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd764132d00 0x7fd7641383d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.453 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 monclient: found mon.noname-c 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd769c90700 1 -- 172.21.15.6:0/4195410516 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd75c009a20 con 0x7fd764098670 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76a491700 10 monclient: get_auth_request con 0x7fd764138d40 auth_method 0 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.440+0000 7fd76a491700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd764138d40 0x7fd76409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.442+0000 7fd769c90700 1 -- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 msgr2=0x7fd764141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.442+0000 7fd769c90700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 0x7fd764141670 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd76000c5d0 tx=0x7fd760007f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7600117a0 con 0x7fd764098670 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:41.454 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: dump: 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.455 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd760011900 con 0x7fd764098670 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 5 monclient: authenticate success, global_id 4817 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 10 monclient: _renew_subs 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd764133650 con 0x7fd764098670 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd760011cc0 con 0x7fd764098670 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd755ffb700 10 monclient: dump: 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.456 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd757fff700 4 set_mon_vals no callback set 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 10 monclient: _renew_subs 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.443+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd76409e0b0 con 0x7fd764098670 2023-09-30T00:49:41.457 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fd760012630 con 0x7fd764098670 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd76bef4700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd76bef4700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd76bef4700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd755ffb700 1 --2- 172.21.15.6:0/4195410516 >> v2:172.21.15.22:6812/142603 conn(0x7fd7480357a0 0x7fd748037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd764072b00 con 0x7fd764098670 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fd7600476f0 con 0x7fd764098670 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd76948f700 1 --2- 172.21.15.6:0/4195410516 >> v2:172.21.15.22:6812/142603 conn(0x7fd7480357a0 0x7fd748037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.458 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.444+0000 7fd76948f700 10 monclient: get_auth_request con 0x7fd7480357a0 auth_method 0 2023-09-30T00:49:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.447+0000 7fd76948f700 1 --2- 172.21.15.6:0/4195410516 >> v2:172.21.15.22:6812/142603 conn(0x7fd7480357a0 0x7fd748037c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd76409dd80 tx=0x7fd758007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.449+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7fd76001c460 con 0x7fd764098670 2023-09-30T00:49:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.449+0000 7fd755ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:41.459 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.449+0000 7fd755ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:41.652 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.652+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7fd764133960 con 0x7fd7480357a0 2023-09-30T00:49:41.653 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.653+0000 7fd755ffb700 1 -- 172.21.15.6:0/4195410516 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fd764133960 con 0x7fd7480357a0 2023-09-30T00:49:41.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 >> v2:172.21.15.22:6812/142603 conn(0x7fd7480357a0 msgr2=0x7fd748037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 --2- 172.21.15.6:0/4195410516 >> v2:172.21.15.22:6812/142603 conn(0x7fd7480357a0 0x7fd748037c50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fd76409dd80 tx=0x7fd758007f90 comp rx=0 tx=0).stop 2023-09-30T00:49:41.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 10 monclient: shutdown 2023-09-30T00:49:41.655 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 msgr2=0x7fd764141670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 0x7fd764141670 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd76000c5d0 tx=0x7fd760007f60 comp rx=0 tx=0).stop 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 shutdown_connections 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 --2- 172.21.15.6:0/4195410516 >> v2:172.21.15.22:6812/142603 conn(0x7fd7480357a0 0x7fd748037c50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fd764138d40 0x7fd76409c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fd764098670 0x7fd764141670 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 --2- 172.21.15.6:0/4195410516 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fd764132d00 0x7fd7641383d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 >> 172.21.15.6:0/4195410516 conn(0x7fd7640a5480 msgr2=0x7fd764130920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 shutdown_connections 2023-09-30T00:49:41.656 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.655+0000 7fd76bef4700 1 -- 172.21.15.6:0/4195410516 wait complete. 2023-09-30T00:49:41.657 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 2 to deep-scrub 2023-09-30T00:49:41.668 INFO:tasks.ceph:Scrubbing osd.3 2023-09-30T00:49:41.668 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.803+0000 7fa5fe4fc700 10 monclient: get_monmap_and_config 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.803+0000 7fa5fe4fc700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.803+0000 7fa5fe4fc700 10 monclient: monmap: 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:41.803772+0000 2023-09-30T00:49:41.803 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:41.803772+0000 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:41.804 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 1 Processor -- start 2023-09-30T00:49:41.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 1 -- start start 2023-09-30T00:49:41.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient: init 2023-09-30T00:49:41.806 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient: _add_conns ranks=[2,0,1,4,3] 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f8098900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient(hunting): picked mon.noname-b con 0x7fa5f8098530 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7fa5f8098e40 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7fa5f8132a70 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f8096af0 con 0x7fa5f8098e40 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa5f8097000 con 0x7fa5f8132a70 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f806a420 con 0x7fa5f8098530 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.804+0000 7fa5fe4fc700 10 monclient(hunting): authenticate will time out at 1578.101898s 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f8098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.807 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f77fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7fa5f8098e40 0x7fa5f8132360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33478/0 (socket says 172.21.15.6:33478) 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f77fe700 1 -- 172.21.15.6:0/4130176073 learned_addr learned my addr 172.21.15.6:0/4130176073 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): get_auth_request con 0x7fa5f8098530 auth_method 0 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_done global_id 4820 payload 274 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 1 -- 172.21.15.6:0/4130176073 >> v1:172.21.15.22:6789/0 conn(0x7fa5f8132a70 legacy=0x7fa5f81394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 1 -- 172.21.15.6:0/4130176073 >> v1:172.21.15.6:6789/0 conn(0x7fa5f8098e40 legacy=0x7fa5f8132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.808 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 1 monclient: found mon.noname-b 2023-09-30T00:49:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.809 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.805+0000 7fa5f7fff700 1 -- 172.21.15.6:0/4130176073 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f8096d20 con 0x7fa5f8098530 2023-09-30T00:49:41.814 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f7fff700 1 -- 172.21.15.6:0/4130176073 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 msgr2=0x7fa5f8098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f7fff700 1 --2- 172.21.15.6:0/4130176073 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f8098900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009910 tx=0x7fa5e80091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a7951463ba9f548 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 1 -- 172.21.15.6:0/4130176073 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800b9f0 con 0x7fa5f8098530 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: dump: 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.815 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5fe4fc700 5 monclient: authenticate success, global_id 4820 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5fe4fc700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 1 -- 172.21.15.6:0/4130176073 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e800bb50 con 0x7fa5f8098530 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 1 -- 172.21.15.6:0/4130176073 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e8011420 con 0x7fa5f8098530 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5fe4fc700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:41.816 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5fe4fc700 4 set_mon_vals no callback set 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.807+0000 7fa5f6ffd700 10 monclient: dump: 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.817 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 10 monclient: shutdown 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/4130176073 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 msgr2=0x7fa5f8098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 --2- 172.21.15.6:0/4130176073 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f8098900 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fa5e8009910 tx=0x7fa5e80091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/4130176073 shutdown_connections 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 --2- 172.21.15.6:0/4130176073 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f8098900 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/4130176073 >> 172.21.15.6:0/4130176073 conn(0x7fa5f80a5460 msgr2=0x7fa5f80a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/4130176073 shutdown_connections 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.808+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/4130176073 wait complete. 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.809+0000 7fa5fe4fc700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.818 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.809+0000 7fa5fe4fc700 10 monclient: monmap: 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.809+0000 7fa5fe4fc700 1 Processor -- start 2023-09-30T00:49:41.819 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 -- start start 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient: init 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f81342d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): picked mon.noname-c con 0x7fa5f8098530 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5f8098e40 0x7fa5f8134810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): picked mon.noname-b con 0x7fa5f8098e40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f8132a70 0x7fa5f8134d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): picked mon.noname-a con 0x7fa5f8132a70 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f8137bc0 con 0x7fa5f8132a70 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.820 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80a11d0 con 0x7fa5f8098e40 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5f80a3360 con 0x7fa5f8098530 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5fe4fc700 10 monclient(hunting): authenticate will time out at 1578.107895s 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5f7fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f81342d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.810+0000 7fa5f7fff700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f81342d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40176/0 (socket says 172.21.15.6:40176) 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 -- 172.21.15.6:0/96303714 learned_addr learned my addr 172.21.15.6:0/96303714 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): get_auth_request con 0x7fa5f8098530 auth_method 0 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.821 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f8132a70 0x7fa5f8134d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): get_auth_request con 0x7fa5f8132a70 auth_method 0 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f77fe700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5f8098e40 0x7fa5f8134810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5fca99700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.822 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f77fe700 10 monclient(hunting): get_auth_request con 0x7fa5f8098e40 auth_method 0 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f77fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f77fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f77fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient(hunting): handle_auth_done global_id 4823 payload 824 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 -- 172.21.15.6:0/96303714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5f8098e40 msgr2=0x7fa5f8134810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5f8098e40 0x7fa5f8134810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 -- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f8132a70 msgr2=0x7fa5f8134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7fa5f8132a70 0x7fa5f8134d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 monclient: found mon.noname-c 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.823 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f7fff700 1 -- 172.21.15.6:0/96303714 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5e8009420 con 0x7fa5f8098530 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.811+0000 7fa5f77fe700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7fa5f8098e40 0x7fa5f8134810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5f7fff700 1 -- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 msgr2=0x7fa5f81342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5f7fff700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f81342d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8097190 tx=0x7fa5e8016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e800bed0 con 0x7fa5f8098530 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: dump: 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.824 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa5e8020070 con 0x7fa5f8098530 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 5 monclient: authenticate success, global_id 4823 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5e801b6a0 con 0x7fa5f8098530 2023-09-30T00:49:41.825 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 10 monclient: _renew_subs 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5f8136960 con 0x7fa5f8098530 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5f67fc700 4 set_mon_vals no callback set 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5dffff700 10 monclient: dump: 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.826 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 10 monclient: _renew_subs 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.813+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5f8135cb0 con 0x7fa5f8098530 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.814+0000 7fa5fe4fc700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.814+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa5f8135cb0 con 0x7fa5f8098530 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.814+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7fa5e8022030 con 0x7fa5f8098530 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.814+0000 7fa5dffff700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6812/142603 conn(0x7fa5e40357a0 0x7fa5e4037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.827 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.814+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7fa5e8049800 con 0x7fa5f8098530 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5dffff700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6800/143135 conn(0x7fa5e403b150 0x7fa5e403d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 --> v2:172.21.15.22:6800/143135 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa5e403dbd0 con 0x7fa5e403b150 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5e804bd30 con 0x7fa5f8098530 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5dffff700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5f77fe700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6812/142603 conn(0x7fa5e40357a0 0x7fa5e4037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5f77fe700 10 monclient: get_auth_request con 0x7fa5e40357a0 auth_method 0 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5fca99700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6800/143135 conn(0x7fa5e403b150 0x7fa5e403d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.828 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.815+0000 7fa5fca99700 10 monclient: get_auth_request con 0x7fa5e403b150 auth_method 0 2023-09-30T00:49:41.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.817+0000 7fa5f77fe700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6812/142603 conn(0x7fa5e40357a0 0x7fa5e4037c50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec007fd0 tx=0x7fa5ec009040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.819+0000 7fa5fca99700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6800/143135 conn(0x7fa5e403b150 0x7fa5e403d560 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.829 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.820+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== osd.3 v2:172.21.15.22:6800/143135 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7fa5e403dbd0 con 0x7fa5e403b150 2023-09-30T00:49:41.834 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.835+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 --> v2:172.21.15.22:6800/143135 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fa5f8072b00 con 0x7fa5e403b150 2023-09-30T00:49:41.835 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.835+0000 7fa5dffff700 1 -- 172.21.15.6:0/96303714 <== osd.3 v2:172.21.15.22:6800/143135 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fa5f8072b00 con 0x7fa5e403b150 2023-09-30T00:49:41.835 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-09-30T00:49:41.835 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:49:41.835 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-09-30T00:49:41.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6800/143135 conn(0x7fa5e403b150 msgr2=0x7fa5e403d560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6800/143135 conn(0x7fa5e403b150 0x7fa5e403d560 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6812/142603 conn(0x7fa5e40357a0 msgr2=0x7fa5e4037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 --2- 172.21.15.6:0/96303714 >> v2:172.21.15.22:6812/142603 conn(0x7fa5e40357a0 0x7fa5e4037c50 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa5ec007fd0 tx=0x7fa5ec009040 comp rx=0 tx=0).stop 2023-09-30T00:49:41.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 10 monclient: shutdown 2023-09-30T00:49:41.836 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 msgr2=0x7fa5f81342d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 --2- 172.21.15.6:0/96303714 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7fa5f8098530 0x7fa5f81342d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fa5f8097190 tx=0x7fa5e8016000 comp rx=0 tx=0).stop 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fca99700 1 -- 172.21.15.6:0/96303714 reap_dead start 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 shutdown_connections 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 >> 172.21.15.6:0/96303714 conn(0x7fa5f80a5460 msgr2=0x7fa5f809b220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 shutdown_connections 2023-09-30T00:49:41.837 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.836+0000 7fa5fe4fc700 1 -- 172.21.15.6:0/96303714 wait complete. 2023-09-30T00:49:41.848 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2023-09-30T00:49:41.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.981+0000 7f58d4049700 10 monclient: get_monmap_and_config 2023-09-30T00:49:41.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.981+0000 7f58d4049700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.981 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.981+0000 7f58d4049700 10 monclient: monmap: 2023-09-30T00:49:41.981 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.981 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:41.982118+0000 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:41.982118+0000 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:41.982 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.982+0000 7f58d4049700 1 Processor -- start 2023-09-30T00:49:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.982+0000 7f58d4049700 1 -- start start 2023-09-30T00:49:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.982+0000 7f58d4049700 10 monclient: init 2023-09-30T00:49:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient: _add_conns ranks=[1,0,2,3,4] 2023-09-30T00:49:41.984 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f58cc098670 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f58cc132d00 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 0x7f58cc13b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient(hunting): picked mon.noname-b con 0x7f58cc138d40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc0a1000 con 0x7f58cc132d00 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58cc0a0730 con 0x7f58cc098670 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc099150 con 0x7f58cc138d40 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d4049700 10 monclient(hunting): authenticate will time out at 1578.280349s 2023-09-30T00:49:41.985 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 0x7f58cc13b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d15e4700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f58cc132d00 0x7f58cc137620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33480/0 (socket says 172.21.15.6:33480) 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d15e4700 1 -- 172.21.15.6:0/4278566612 learned_addr learned my addr 172.21.15.6:0/4278566612 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): get_auth_request con 0x7f58cc138d40 auth_method 0 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:41.986 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient(hunting): handle_auth_done global_id 4826 payload 274 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 1 -- 172.21.15.6:0/4278566612 >> v1:172.21.15.22:6789/0 conn(0x7f58cc098670 legacy=0x7f58cc098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 1 -- 172.21.15.6:0/4278566612 >> v1:172.21.15.6:6789/0 conn(0x7f58cc132d00 legacy=0x7f58cc137620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient: _finish_hunting 0 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 1 monclient: found mon.noname-b 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:41.987 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.983+0000 7f58d25e6700 1 -- 172.21.15.6:0/4278566612 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc068df0 con 0x7f58cc138d40 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d25e6700 1 -- 172.21.15.6:0/4278566612 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 msgr2=0x7f58cc13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d25e6700 1 --2- 172.21.15.6:0/4278566612 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 0x7f58cc13b120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009910 tx=0x7f58c40091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=317552ee555aba79 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 1 -- 172.21.15.6:0/4278566612 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c400b9f0 con 0x7f58cc138d40 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: dump: 2023-09-30T00:49:41.992 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.993 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: _finish_auth 0 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 1 -- 172.21.15.6:0/4278566612 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58c400bb50 con 0x7f58cc138d40 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 5 monclient: authenticate success, global_id 4826 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 1 -- 172.21.15.6:0/4278566612 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58c4011420 con 0x7f58cc138d40 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:41.994 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 4 set_mon_vals no callback set 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d0de3700 10 monclient: dump: 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:41.995 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 10 monclient: shutdown 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 1 -- 172.21.15.6:0/4278566612 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 msgr2=0x7f58cc13b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 1 --2- 172.21.15.6:0/4278566612 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 0x7f58cc13b120 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f58c4009910 tx=0x7f58c40091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 1 -- 172.21.15.6:0/4278566612 shutdown_connections 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 1 --2- 172.21.15.6:0/4278566612 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc138d40 0x7f58cc13b120 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:41.996 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 1 -- 172.21.15.6:0/4278566612 >> 172.21.15.6:0/4278566612 conn(0x7f58cc0a5480 msgr2=0x7f58cc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.986+0000 7f58d4049700 1 -- 172.21.15.6:0/4278566612 shutdown_connections 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.987+0000 7f58d4049700 1 -- 172.21.15.6:0/4278566612 wait complete. 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.987+0000 7f58d4049700 10 monclient: build_initial_monmap 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.987+0000 7f58d4049700 10 monclient: monmap: 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:41.997 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.987+0000 7f58d4049700 1 Processor -- start 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.987+0000 7f58d4049700 1 -- start start 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.987+0000 7f58d4049700 10 monclient: init 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:49:41.998 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 0x7f58cc141640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): picked mon.noname-a con 0x7f58cc098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc132d00 0x7f58cc1383c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): picked mon.noname-c con 0x7f58cc132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58cc138d40 0x7f58cc13f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): picked mon.noname-b con 0x7f58cc138d40 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc12e0e0 con 0x7f58cc098670 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc12f160 con 0x7f58cc138d40 2023-09-30T00:49:41.999 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58cc1335c0 con 0x7f58cc132d00 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d4049700 10 monclient(hunting): authenticate will time out at 1578.285437s 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 0x7f58cc141640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc132d00 0x7f58cc1383c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 0x7f58cc141640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35848/0 (socket says 172.21.15.6:35848) 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 -- 172.21.15.6:0/2274382510 learned_addr learned my addr 172.21.15.6:0/2274382510 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): get_auth_request con 0x7f58cc098670 auth_method 0 2023-09-30T00:49:42.000 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): get_auth_request con 0x7f58cc132d00 auth_method 0 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.001 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.002 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d15e4700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d25e6700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58cc138d40 0x7f58cc13f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.003 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d25e6700 10 monclient(hunting): get_auth_request con 0x7f58cc138d40 auth_method 0 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d25e6700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d25e6700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d25e6700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient(hunting): handle_auth_done global_id 4776 payload 824 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 -- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc132d00 msgr2=0x7f58cc1383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc132d00 0x7f58cc1383c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 -- 172.21.15.6:0/2274382510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58cc138d40 msgr2=0x7f58cc13f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58cc138d40 0x7f58cc13f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient: _finish_hunting 0 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 monclient: found mon.noname-a 2023-09-30T00:49:42.004 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d1de5700 1 -- 172.21.15.6:0/2274382510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c4009420 con 0x7f58cc098670 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.988+0000 7f58d25e6700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58cc138d40 0x7f58cc13f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d1de5700 1 -- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 msgr2=0x7f58cc141640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d1de5700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 0x7f58cc141640 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c950 tx=0x7f58bc002760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc012840 con 0x7f58cc098670 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: dump: 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.005 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:42.006 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58bc0129a0 con 0x7f58cc098670 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 5 monclient: authenticate success, global_id 4776 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 10 monclient: _renew_subs 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58cc1338a0 con 0x7f58cc098670 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58bc012d60 con 0x7f58cc098670 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:42.007 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58c9ffb700 10 monclient: dump: 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.008 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 10 monclient: _renew_subs 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58cbfff700 4 set_mon_vals no callback set 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.991+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58cc12e520 con 0x7f58cc098670 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f58bc00a630 con 0x7f58cc098670 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58d4049700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58d4049700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58d4049700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.009 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58cc133dc0 con 0x7f58cc098670 2023-09-30T00:49:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58c9ffb700 1 --2- 172.21.15.6:0/2274382510 >> v2:172.21.15.22:6812/142603 conn(0x7f58b80357a0 0x7f58b8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f58bc047e60 con 0x7f58cc098670 2023-09-30T00:49:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58d15e4700 1 --2- 172.21.15.6:0/2274382510 >> v2:172.21.15.22:6812/142603 conn(0x7f58b80357a0 0x7f58b8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.992+0000 7f58d15e4700 10 monclient: get_auth_request con 0x7f58b80357a0 auth_method 0 2023-09-30T00:49:42.010 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.995+0000 7f58d15e4700 1 --2- 172.21.15.6:0/2274382510 >> v2:172.21.15.22:6812/142603 conn(0x7f58b80357a0 0x7f58b8037c50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f58cc1426d0 tx=0x7f58c0007f90 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.997+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f58cc133dc0 con 0x7f58cc098670 2023-09-30T00:49:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.997+0000 7f58c9ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:42.011 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:41.997+0000 7f58c9ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:42.198 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.199+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7f58cc133d90 con 0x7f58b80357a0 2023-09-30T00:49:42.199 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.199+0000 7f58c9ffb700 1 -- 172.21.15.6:0/2274382510 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f58cc133d90 con 0x7f58b80357a0 2023-09-30T00:49:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.201+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 >> v2:172.21.15.22:6812/142603 conn(0x7f58b80357a0 msgr2=0x7f58b8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.201 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 --2- 172.21.15.6:0/2274382510 >> v2:172.21.15.22:6812/142603 conn(0x7f58b80357a0 0x7f58b8037c50 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f58cc1426d0 tx=0x7f58c0007f90 comp rx=0 tx=0).stop 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 10 monclient: shutdown 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 msgr2=0x7f58cc141640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 0x7f58cc141640 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f58bc00c950 tx=0x7f58bc002760 comp rx=0 tx=0).stop 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 shutdown_connections 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 --2- 172.21.15.6:0/2274382510 >> v2:172.21.15.22:6812/142603 conn(0x7f58b80357a0 0x7f58b8037c50 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f58cc138d40 0x7f58cc13f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f58cc098670 0x7f58cc141640 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 --2- 172.21.15.6:0/2274382510 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f58cc132d00 0x7f58cc1383c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 >> 172.21.15.6:0/2274382510 conn(0x7f58cc0a5480 msgr2=0x7f58cc0975f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:42.202 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 shutdown_connections 2023-09-30T00:49:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.202+0000 7f58d4049700 1 -- 172.21.15.6:0/2274382510 wait complete. 2023-09-30T00:49:42.203 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 3 to deep-scrub 2023-09-30T00:49:42.215 INFO:tasks.ceph:Scrubbing osd.4 2023-09-30T00:49:42.215 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:49:42.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.353+0000 7f5624e8d700 10 monclient: get_monmap_and_config 2023-09-30T00:49:42.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.353+0000 7f5624e8d700 10 monclient: build_initial_monmap 2023-09-30T00:49:42.355 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.353+0000 7f5624e8d700 10 monclient: monmap: 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:42.353486+0000 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:42.353486+0000 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:42.356 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.353+0000 7f5624e8d700 1 Processor -- start 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.353+0000 7f5624e8d700 1 -- start start 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.353+0000 7f5624e8d700 10 monclient: init 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient: _add_conns ranks=[2,0,3,1,4] 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 0x7f5620098900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): picked mon.noname-b con 0x7f5620098530 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f5620098e40 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5620132a70 0x7f56201394b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): picked mon.noname-a con 0x7f5620132a70 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:42.357 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5620096af0 con 0x7f5620098e40 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f562006a420 con 0x7f5620132a70 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5620097000 con 0x7f5620098530 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f5624e8d700 10 monclient(hunting): authenticate will time out at 1578.651660s 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f5620132a70 0x7f56201394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 0x7f5620098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561dd9b700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f5620098e40 0x7f5620132360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33492/0 (socket says 172.21.15.6:33492) 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561dd9b700 1 -- 172.21.15.6:0/38827273 learned_addr learned my addr 172.21.15.6:0/38827273 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): get_auth_request con 0x7f5620132a70 auth_method 0 2023-09-30T00:49:42.358 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): get_auth_request con 0x7f5620098530 auth_method 0 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.359 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.354+0000 7f561e59c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 10 monclient(hunting): handle_auth_done global_id 4832 payload 274 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 1 -- 172.21.15.6:0/38827273 >> v2:172.21.15.6:3300/0 conn(0x7f5620132a70 msgr2=0x7f56201394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 1 --2- 172.21.15.6:0/38827273 >> v2:172.21.15.6:3300/0 conn(0x7f5620132a70 0x7f56201394b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 1 -- 172.21.15.6:0/38827273 >> v1:172.21.15.6:6789/0 conn(0x7f5620098e40 legacy=0x7f5620132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 10 monclient: _finish_hunting 0 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 1 monclient: found mon.noname-b 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561e59c700 1 -- 172.21.15.6:0/38827273 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5620096d20 con 0x7f5620098530 2023-09-30T00:49:42.360 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.355+0000 7f561ed9d700 1 --2- 172.21.15.6:0/38827273 >> v2:172.21.15.6:3300/0 conn(0x7f5620132a70 0x7f56201394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561e59c700 1 -- 172.21.15.6:0/38827273 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 msgr2=0x7f5620098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561e59c700 1 --2- 172.21.15.6:0/38827273 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 0x7f5620098900 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f56100098e0 tx=0x7f56100091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6643f7edd785e44b server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 1 -- 172.21.15.6:0/38827273 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610011840 con 0x7f5620098530 2023-09-30T00:49:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.366 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: dump: 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: _finish_auth 0 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 1 -- 172.21.15.6:0/38827273 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f56100119a0 con 0x7f5620098530 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 1 -- 172.21.15.6:0/38827273 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5610011d40 con 0x7f5620098530 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f5624e8d700 5 monclient: authenticate success, global_id 4832 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f5624e8d700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f5624e8d700 4 set_mon_vals no callback set 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.357+0000 7f561d59a700 10 monclient: dump: 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.368 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 10 monclient: shutdown 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 -- 172.21.15.6:0/38827273 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 msgr2=0x7f5620098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 --2- 172.21.15.6:0/38827273 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 0x7f5620098900 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f56100098e0 tx=0x7f56100091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 -- 172.21.15.6:0/38827273 shutdown_connections 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 --2- 172.21.15.6:0/38827273 >> v2:172.21.15.6:3300/0 conn(0x7f5620132a70 0x7f56201394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 --2- 172.21.15.6:0/38827273 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098530 0x7f5620098900 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 -- 172.21.15.6:0/38827273 >> 172.21.15.6:0/38827273 conn(0x7f56200a5460 msgr2=0x7f56200a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 -- 172.21.15.6:0/38827273 shutdown_connections 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 1 -- 172.21.15.6:0/38827273 wait complete. 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 10 monclient: build_initial_monmap 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.358+0000 7f5624e8d700 10 monclient: monmap: 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:42.370 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 Processor -- start 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 -- start start 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient: init 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient: _add_conns ranks=[1,2,0] 2023-09-30T00:49:42.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5620098530 0x7f5620133270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): picked mon.noname-b con 0x7f5620098530 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098e40 0x7f56201337b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): picked mon.noname-c con 0x7f5620098e40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 0x7f5620133cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): picked mon.noname-a con 0x7f5620132a70 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200a11d0 con 0x7f5620132a70 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56200a3360 con 0x7f5620098530 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5620136500 con 0x7f5620098e40 2023-09-30T00:49:42.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f5624e8d700 10 monclient(hunting): authenticate will time out at 1578.656853s 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f561ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 0x7f5620133cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.359+0000 7f561dd9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098e40 0x7f56201337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 0x7f5620133cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35854/0 (socket says 172.21.15.6:35854) 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 -- 172.21.15.6:0/649077635 learned_addr learned my addr 172.21.15.6:0/649077635 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): get_auth_request con 0x7f5620132a70 auth_method 0 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098e40 0x7f56201337b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40212/0 (socket says 172.21.15.6:40212) 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): get_auth_request con 0x7f5620098e40 auth_method 0 2023-09-30T00:49:42.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561dd9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561e59c700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5620098530 0x7f5620133270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561e59c700 10 monclient(hunting): get_auth_request con 0x7f5620098530 auth_method 0 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561e59c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561e59c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561e59c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient(hunting): handle_auth_done global_id 4782 payload 824 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 -- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098e40 msgr2=0x7f56201337b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f5620098e40 0x7f56201337b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 -- 172.21.15.6:0/649077635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5620098530 msgr2=0x7f5620133270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5620098530 0x7f5620133270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient: _finish_hunting 0 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 monclient: found mon.noname-a 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561ed9d700 1 -- 172.21.15.6:0/649077635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5618009a20 con 0x7f5620132a70 2023-09-30T00:49:42.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.360+0000 7f561e59c700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f5620098530 0x7f5620133270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f561ed9d700 1 -- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 msgr2=0x7f5620133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f561ed9d700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 0x7f5620133cf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5618009f90 tx=0x7f561800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f561800ce90 con 0x7f5620132a70 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: dump: 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.376 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f561800f040 con 0x7f5620132a70 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 5 monclient: authenticate success, global_id 4782 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 10 monclient: _renew_subs 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610009420 con 0x7f5620132a70 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56180146b0 con 0x7f5620132a70 2023-09-30T00:49:42.377 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f560e7fc700 10 monclient: dump: 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.378 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f561cd99700 4 set_mon_vals no callback set 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 10 monclient: _renew_subs 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.362+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5620134610 con 0x7f5620132a70 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.363+0000 7f5624e8d700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.363+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f5620072b00 con 0x7f5620132a70 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.363+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f5618014a20 con 0x7f5620132a70 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.363+0000 7f560e7fc700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6812/142603 conn(0x7f56000357a0 0x7f5600037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.363+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f5618048770 con 0x7f5620132a70 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f560e7fc700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6808/143134 conn(0x7f560003b150 0x7f560003d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 --> v2:172.21.15.22:6808/143134 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f560003dbd0 con 0x7f560003b150 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f561804ad00 con 0x7f5620132a70 2023-09-30T00:49:42.379 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f560e7fc700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:49:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f561e59c700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6812/142603 conn(0x7f56000357a0 0x7f5600037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f561e59c700 10 monclient: get_auth_request con 0x7f56000357a0 auth_method 0 2023-09-30T00:49:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f561dd9b700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6808/143134 conn(0x7f560003b150 0x7f560003d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.364+0000 7f561dd9b700 10 monclient: get_auth_request con 0x7f560003b150 auth_method 0 2023-09-30T00:49:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.366+0000 7f561e59c700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6812/142603 conn(0x7f56000357a0 0x7f5600037c50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f56100030c0 tx=0x7f5610016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.380 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.369+0000 7f561dd9b700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6808/143134 conn(0x7f560003b150 0x7f560003d560 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.369+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== osd.4 v2:172.21.15.22:6808/143134 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f560003dbd0 con 0x7f560003b150 2023-09-30T00:49:42.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.384+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 --> v2:172.21.15.22:6808/143134 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f562009f460 con 0x7f560003b150 2023-09-30T00:49:42.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.384+0000 7f560e7fc700 1 -- 172.21.15.6:0/649077635 <== osd.4 v2:172.21.15.22:6808/143134 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f562009f460 con 0x7f560003b150 2023-09-30T00:49:42.384 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-09-30T00:49:42.384 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:49:42.384 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-09-30T00:49:42.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6808/143134 conn(0x7f560003b150 msgr2=0x7f560003d560 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6808/143134 conn(0x7f560003b150 0x7f560003d560 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6812/142603 conn(0x7f56000357a0 msgr2=0x7f5600037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 --2- 172.21.15.6:0/649077635 >> v2:172.21.15.22:6812/142603 conn(0x7f56000357a0 0x7f5600037c50 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f56100030c0 tx=0x7f5610016000 comp rx=0 tx=0).stop 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 10 monclient: shutdown 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 msgr2=0x7f5620133cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 --2- 172.21.15.6:0/649077635 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f5620132a70 0x7f5620133cf0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f5618009f90 tx=0x7f561800b040 comp rx=0 tx=0).stop 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f561ed9d700 1 -- 172.21.15.6:0/649077635 reap_dead start 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 shutdown_connections 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 >> 172.21.15.6:0/649077635 conn(0x7f56200a5460 msgr2=0x7f5620068d10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 shutdown_connections 2023-09-30T00:49:42.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.385+0000 7f5624e8d700 1 -- 172.21.15.6:0/649077635 wait complete. 2023-09-30T00:49:42.397 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.529+0000 7f0b61e71700 10 monclient: get_monmap_and_config 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.529+0000 7f0b61e71700 10 monclient: build_initial_monmap 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.529+0000 7f0b61e71700 10 monclient: monmap: 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:42.529677+0000 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:42.529677+0000 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:42.529 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:42.530 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:42.530 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 Processor -- start 2023-09-30T00:49:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 -- start start 2023-09-30T00:49:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient: init 2023-09-30T00:49:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:42.532 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient: _add_conns ranks=[4,2,3,0,1] 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f0b5c098670 0x7f0b5c132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): picked mon.noname-c con 0x7f0b5c098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 0x7f0b5c12ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 0x7f0b5c12f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c098e90 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5c0a0a50 con 0x7f0b5c098e90 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b5c0a1000 con 0x7f0b5c098670 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.533 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c0a0730 con 0x7f0b5c098a40 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b61e71700 10 monclient(hunting): authenticate will time out at 1578.827811s 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b5bfff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 0x7f0b5c12f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.530+0000 7f0b5affd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 0x7f0b5c12ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 0x7f0b5c12f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35856/0 (socket says 172.21.15.6:35856) 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 1 -- 172.21.15.6:0/836886733 learned_addr learned my addr 172.21.15.6:0/836886733 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): get_auth_request con 0x7f0b5c098a40 auth_method 0 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): get_auth_request con 0x7f0b5c098e90 auth_method 0 2023-09-30T00:49:42.534 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5b7fe700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.22:3300/0 conn(0x7f0b5c098670 0x7f0b5c132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5b7fe700 10 monclient(hunting): get_auth_request con 0x7f0b5c098670 auth_method 0 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.535 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient(hunting): handle_auth_done global_id 4838 payload 274 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 1 -- 172.21.15.6:0/836886733 >> v2:172.21.15.22:3300/0 conn(0x7f0b5c098670 msgr2=0x7f0b5c132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.22:3300/0 conn(0x7f0b5c098670 0x7f0b5c132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 1 -- 172.21.15.6:0/836886733 >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 msgr2=0x7f0b5c12f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 0x7f0b5c12f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 1 monclient: found mon.noname-b 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5affd700 1 -- 172.21.15.6:0/836886733 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c068df0 con 0x7f0b5c098a40 2023-09-30T00:49:42.536 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5bfff700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 0x7f0b5c12f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:42.537 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.531+0000 7f0b5b7fe700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.22:3300/0 conn(0x7f0b5c098670 0x7f0b5c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5affd700 1 -- 172.21.15.6:0/836886733 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 msgr2=0x7f0b5c12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5affd700 1 --2- 172.21.15.6:0/836886733 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 0x7f0b5c12ede0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c09fe60 tx=0x7f0b500091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4aacd0ea0507833 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 1 -- 172.21.15.6:0/836886733 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011840 con 0x7f0b5c098a40 2023-09-30T00:49:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:42.543 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: dump: 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: _finish_auth 0 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:42.544 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b61e71700 5 monclient: authenticate success, global_id 4838 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 1 -- 172.21.15.6:0/836886733 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b500119a0 con 0x7f0b5c098a40 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b61e71700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b61e71700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b61e71700 4 set_mon_vals no callback set 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 1 -- 172.21.15.6:0/836886733 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b50011d40 con 0x7f0b5c098a40 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.533+0000 7f0b5a7fc700 10 monclient: dump: 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.545 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 10 monclient: shutdown 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 -- 172.21.15.6:0/836886733 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 msgr2=0x7f0b5c12ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 --2- 172.21.15.6:0/836886733 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 0x7f0b5c12ede0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c09fe60 tx=0x7f0b500091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.546 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 -- 172.21.15.6:0/836886733 shutdown_connections 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.6:3300/0 conn(0x7f0b5c098e90 0x7f0b5c12f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 --2- 172.21.15.6:0/836886733 >> v2:172.21.15.22:3300/0 conn(0x7f0b5c098670 0x7f0b5c132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 --2- 172.21.15.6:0/836886733 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098a40 0x7f0b5c12ede0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 -- 172.21.15.6:0/836886733 >> 172.21.15.6:0/836886733 conn(0x7f0b5c0a5480 msgr2=0x7f0b5c0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 -- 172.21.15.6:0/836886733 shutdown_connections 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.534+0000 7f0b61e71700 1 -- 172.21.15.6:0/836886733 wait complete. 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.535+0000 7f0b61e71700 10 monclient: build_initial_monmap 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.535+0000 7f0b61e71700 10 monclient: monmap: 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:42.547 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.535+0000 7f0b61e71700 1 Processor -- start 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.535+0000 7f0b61e71700 1 -- start start 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.535+0000 7f0b61e71700 10 monclient: init 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.548 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): picked mon.noname-c con 0x7f0b5c098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 0x7f0b5c12d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): picked mon.noname-a con 0x7f0b5c098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0b5c098e90 0x7f0b5c12d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): picked mon.noname-b con 0x7f0b5c098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c12de00 con 0x7f0b5c098a40 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c12f6c0 con 0x7f0b5c098e90 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c12e460 con 0x7f0b5c098670 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b61e71700 10 monclient(hunting): authenticate will time out at 1578.833224s 2023-09-30T00:49:42.549 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 0x7f0b5c12d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40226/0 (socket says 172.21.15.6:40226) 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 0x7f0b5c12d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35860/0 (socket says 172.21.15.6:35860) 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 -- 172.21.15.6:0/2089846318 learned_addr learned my addr 172.21.15.6:0/2089846318 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): get_auth_request con 0x7f0b5c098a40 auth_method 0 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): get_auth_request con 0x7f0b5c098670 auth_method 0 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.550 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5bfff700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0b5c098e90 0x7f0b5c12d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5bfff700 10 monclient(hunting): get_auth_request con 0x7f0b5c098e90 auth_method 0 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5bfff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5bfff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5bfff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:42.551 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient(hunting): handle_auth_done global_id 4841 payload 824 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 -- 172.21.15.6:0/2089846318 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0b5c098e90 msgr2=0x7f0b5c12d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0b5c098e90 0x7f0b5c12d7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 -- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 msgr2=0x7f0b5c12d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 0x7f0b5c12d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 monclient: found mon.noname-c 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5b7fe700 1 -- 172.21.15.6:0/2089846318 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b54009ae0 con 0x7f0b5c098670 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5bfff700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0b5c098e90 0x7f0b5c12d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.536+0000 7f0b5affd700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 0x7f0b5c12d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b5b7fe700 1 -- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 msgr2=0x7f0b5c12cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:42.552 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b5b7fe700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00e8d0 tx=0x7f0b4c008040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c013620 con 0x7f0b5c098670 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: dump: 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.553 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: _finish_auth 0 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b4c013780 con 0x7f0b5c098670 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 5 monclient: authenticate success, global_id 4841 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 10 monclient: _renew_subs 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b50009420 con 0x7f0b5c098670 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4c013b20 con 0x7f0b5c098670 2023-09-30T00:49:42.554 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:42.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:42.555 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b437fe700 10 monclient: dump: 2023-09-30T00:49:42.555 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:42.555 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:42.555 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.555 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:42.557 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b59ffb700 4 set_mon_vals no callback set 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 10 monclient: _renew_subs 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.539+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c09eb50 con 0x7f0b5c098670 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b5b7fe700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00e8d0 tx=0x7f0b4c008040 comp rx=0 tx=0).queue_received will delay after 1.69603e+09 on 0x7f0b4c00c3a0 mgrmap(e 5) v1 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f0b4c00c3a0 con 0x7f0b5c098670 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b61e71700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b61e71700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b61e71700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b5c0a13f0 con 0x7f0b5c098670 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b437fe700 1 --2- 172.21.15.6:0/2089846318 >> v2:172.21.15.22:6812/142603 conn(0x7f0b480357a0 0x7f0b48037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b5affd700 1 --2- 172.21.15.6:0/2089846318 >> v2:172.21.15.22:6812/142603 conn(0x7f0b480357a0 0x7f0b48037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:42.558 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.540+0000 7f0b5affd700 10 monclient: get_auth_request con 0x7f0b480357a0 auth_method 0 2023-09-30T00:49:42.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.543+0000 7f0b5affd700 1 --2- 172.21.15.6:0/2089846318 >> v2:172.21.15.22:6812/142603 conn(0x7f0b480357a0 0x7f0b48037c50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c09f080 tx=0x7f0b50017000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:42.559 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:42.545+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f0b4c047790 con 0x7f0b5c098670 2023-09-30T00:49:43.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.368+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f0b4c022c90 con 0x7f0b5c098670 2023-09-30T00:49:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.368+0000 7f0b437fe700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:43.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.368+0000 7f0b437fe700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:43.572 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.573+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b5c12f800 con 0x7f0b480357a0 2023-09-30T00:49:43.573 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.573+0000 7f0b437fe700 1 -- 172.21.15.6:0/2089846318 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0b50000c50 con 0x7f0b480357a0 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 >> v2:172.21.15.22:6812/142603 conn(0x7f0b480357a0 msgr2=0x7f0b48037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 --2- 172.21.15.6:0/2089846318 >> v2:172.21.15.22:6812/142603 conn(0x7f0b480357a0 0x7f0b48037c50 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f0b5c09f080 tx=0x7f0b50017000 comp rx=0 tx=0).stop 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 10 monclient: shutdown 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 msgr2=0x7f0b5c12cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c00e8d0 tx=0x7f0b4c008040 comp rx=0 tx=0).stop 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 shutdown_connections 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 --2- 172.21.15.6:0/2089846318 >> v2:172.21.15.22:6812/142603 conn(0x7f0b480357a0 0x7f0b48037c50 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.576 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f0b5c098e90 0x7f0b5c12d7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f0b5c098670 0x7f0b5c12cd50 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 --2- 172.21.15.6:0/2089846318 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f0b5c098a40 0x7f0b5c12d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 >> 172.21.15.6:0/2089846318 conn(0x7f0b5c0a5480 msgr2=0x7f0b5c069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 shutdown_connections 2023-09-30T00:49:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.576+0000 7f0b61e71700 1 -- 172.21.15.6:0/2089846318 wait complete. 2023-09-30T00:49:43.577 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 4 to deep-scrub 2023-09-30T00:49:43.589 INFO:tasks.ceph:Scrubbing osd.5 2023-09-30T00:49:43.589 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2023-09-30T00:49:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.724+0000 7f255e57c700 10 monclient: get_monmap_and_config 2023-09-30T00:49:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.724+0000 7f255e57c700 10 monclient: build_initial_monmap 2023-09-30T00:49:43.724 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.724+0000 7f255e57c700 10 monclient: monmap: 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:43.725030+0000 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:43.725030+0000 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:43.725 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.725+0000 7f255e57c700 1 Processor -- start 2023-09-30T00:49:43.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.725+0000 7f255e57c700 1 -- start start 2023-09-30T00:49:43.728 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.725+0000 7f255e57c700 10 monclient: init 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient: _add_conns ranks=[3,0,2,1,4] 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2558098530 0x7f2558098900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): picked mon.noname-a con 0x7f2558098530 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f2558098e40 addr v1:172.21.15.6:6789/0 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558132a70 0x7f25581394b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): picked mon.noname-b con 0x7f2558132a70 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2558096af0 con 0x7f2558098e40 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f255806a420 con 0x7f2558098530 2023-09-30T00:49:43.729 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2558097000 con 0x7f2558132a70 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255e57c700 10 monclient(hunting): authenticate will time out at 1580.023302s 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f2558098530 0x7f2558098900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f25577fe700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f2558098e40 0x7f2558132360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:33496/0 (socket says 172.21.15.6:33496) 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558132a70 0x7f25581394b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f25577fe700 1 -- 172.21.15.6:0/3394486454 learned_addr learned my addr 172.21.15.6:0/3394486454 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): get_auth_request con 0x7f2558098530 auth_method 0 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.730 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): get_auth_request con 0x7f2558132a70 auth_method 0 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.731 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient(hunting): handle_auth_done global_id 4791 payload 274 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 1 -- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558132a70 msgr2=0x7f25581394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 1 --2- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558132a70 0x7f25581394b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 1 -- 172.21.15.6:0/3394486454 >> v1:172.21.15.6:6789/0 conn(0x7f2558098e40 legacy=0x7f2558132360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient: _finish_hunting 0 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 1 monclient: found mon.noname-a 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f2557fff700 1 -- 172.21.15.6:0/3394486454 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2558096d20 con 0x7f2558098530 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.726+0000 7f255cb19700 1 --2- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558132a70 0x7f25581394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2557fff700 1 -- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098530 msgr2=0x7f2558098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:43.732 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2557fff700 1 --2- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098530 0x7f2558098900 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f25480098e0 tx=0x7f25480091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ee7ff41c6b04916 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 1 -- 172.21.15.6:0/3394486454 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548011840 con 0x7f2558098530 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: dump: 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.733 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 5 monclient: authenticate success, global_id 4791 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 1 -- 172.21.15.6:0/3394486454 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25480119a0 con 0x7f2558098530 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 4 set_mon_vals no callback set 2023-09-30T00:49:43.734 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 1 -- 172.21.15.6:0/3394486454 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2548011d40 con 0x7f2558098530 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f2556ffd700 10 monclient: dump: 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.735 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 10 monclient: shutdown 2023-09-30T00:49:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 -- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098530 msgr2=0x7f2558098900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 --2- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098530 0x7f2558098900 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f25480098e0 tx=0x7f25480091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.740 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 -- 172.21.15.6:0/3394486454 shutdown_connections 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 --2- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558132a70 0x7f25581394b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 --2- 172.21.15.6:0/3394486454 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098530 0x7f2558098900 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 -- 172.21.15.6:0/3394486454 >> 172.21.15.6:0/3394486454 conn(0x7f25580a5460 msgr2=0x7f25580a5860 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 -- 172.21.15.6:0/3394486454 shutdown_connections 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.729+0000 7f255e57c700 1 -- 172.21.15.6:0/3394486454 wait complete. 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.730+0000 7f255e57c700 10 monclient: build_initial_monmap 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.730+0000 7f255e57c700 10 monclient: monmap: 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:43.741 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.730+0000 7f255e57c700 1 Processor -- start 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 -- start start 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient: init 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558098530 0x7f25581343e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): picked mon.noname-c con 0x7f2558098530 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:43.742 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 0x7f2558134920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): picked mon.noname-a con 0x7f2558098e40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2558132a70 0x7f2558134e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): picked mon.noname-b con 0x7f2558132a70 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25581378c0 con 0x7f2558098e40 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25580a3360 con 0x7f2558132a70 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2558138070 con 0x7f2558098530 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255e57c700 10 monclient(hunting): authenticate will time out at 1580.028517s 2023-09-30T00:49:43.743 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 0x7f2558134920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 0x7f2558134920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35870/0 (socket says 172.21.15.6:35870) 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 -- 172.21.15.6:0/537161246 learned_addr learned my addr 172.21.15.6:0/537161246 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): get_auth_request con 0x7f2558098e40 auth_method 0 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558098530 0x7f25581343e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): get_auth_request con 0x7f2558098530 auth_method 0 2023-09-30T00:49:43.744 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f2557fff700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient(hunting): handle_auth_done global_id 4794 payload 824 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255cb19700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2558132a70 0x7f2558134e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 -- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558098530 msgr2=0x7f25581343e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f2558098530 0x7f25581343e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 -- 172.21.15.6:0/537161246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2558132a70 msgr2=0x7f2558134e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2558132a70 0x7f2558134e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 monclient: found mon.noname-a 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:43.746 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f25577fe700 1 -- 172.21.15.6:0/537161246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2550009a20 con 0x7f2558098e40 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255cb19700 10 monclient: get_auth_request con 0x7f2558132a70 auth_method 0 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.731+0000 7f255cb19700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f2558132a70 0x7f2558134e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f25577fe700 1 -- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 msgr2=0x7f2558134920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f25577fe700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 0x7f2558134920 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f254000c5e0 tx=0x7f2540004a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540005d40 con 0x7f2558098e40 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: dump: 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.747 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: _finish_auth 0 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25400070d0 con 0x7f2558098e40 2023-09-30T00:49:43.748 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 5 monclient: authenticate success, global_id 4794 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 10 monclient: _renew_subs 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2548009420 con 0x7f2558098e40 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2540007490 con 0x7f2558098e40 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f254ffff700 10 monclient: dump: 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.749 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f25567fc700 4 set_mon_vals no callback set 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 10 monclient: _renew_subs 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.734+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25581353f0 con 0x7f2558098e40 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f255e57c700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2558072b00 con 0x7f2558098e40 2023-09-30T00:49:43.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f254001b3b0 con 0x7f2558098e40 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6812/142603 conn(0x7f25380357a0 0x7f2538037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f25400475c0 con 0x7f2558098e40 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6804/143133 conn(0x7f253803b2c0 0x7f253803d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 --> v2:172.21.15.22:6804/143133 -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f253803dd40 con 0x7f253803b2c0 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_get_version_reply(handle=1 version=17) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2540047860 con 0x7f2558098e40 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.735+0000 7f254ffff700 10 monclient: handle_get_version_reply finishing 1 version 17 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.736+0000 7f2557fff700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6812/142603 conn(0x7f25380357a0 0x7f2538037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.736+0000 7f2557fff700 10 monclient: get_auth_request con 0x7f25380357a0 auth_method 0 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.736+0000 7f255cb19700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6804/143133 conn(0x7f253803b2c0 0x7f253803d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.736+0000 7f255cb19700 10 monclient: get_auth_request con 0x7f253803b2c0 auth_method 0 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.738+0000 7f2557fff700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6812/142603 conn(0x7f25380357a0 0x7f2538037c50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f25480030c0 tx=0x7f2548016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.740+0000 7f255cb19700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6804/143133 conn(0x7f253803b2c0 0x7f253803d6d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.741+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== osd.5 v2:172.21.15.22:6804/143133 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+23999 (crc 0 0 0) 0x7f253803dd40 con 0x7f253803b2c0 2023-09-30T00:49:43.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.756+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 --> v2:172.21.15.22:6804/143133 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f2558136bc0 con 0x7f253803b2c0 2023-09-30T00:49:43.756 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f254ffff700 1 -- 172.21.15.6:0/537161246 <== osd.5 v2:172.21.15.22:6804/143133 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f2558136bc0 con 0x7f253803b2c0 2023-09-30T00:49:43.756 INFO:teuthology.orchestra.run.smithi006.stdout:{ 2023-09-30T00:49:43.757 INFO:teuthology.orchestra.run.smithi006.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2023-09-30T00:49:43.757 INFO:teuthology.orchestra.run.smithi006.stdout:} 2023-09-30T00:49:43.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6804/143133 conn(0x7f253803b2c0 msgr2=0x7f253803d6d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6804/143133 conn(0x7f253803b2c0 0x7f253803d6d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6812/142603 conn(0x7f25380357a0 msgr2=0x7f2538037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 --2- 172.21.15.6:0/537161246 >> v2:172.21.15.22:6812/142603 conn(0x7f25380357a0 0x7f2538037c50 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f25480030c0 tx=0x7f2548016000 comp rx=0 tx=0).stop 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 10 monclient: shutdown 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 msgr2=0x7f2558134920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 --2- 172.21.15.6:0/537161246 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f2558098e40 0x7f2558134920 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f254000c5e0 tx=0x7f2540004a80 comp rx=0 tx=0).stop 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255cb19700 1 -- 172.21.15.6:0/537161246 reap_dead start 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 shutdown_connections 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 >> 172.21.15.6:0/537161246 conn(0x7f25580a5460 msgr2=0x7f2558068dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 shutdown_connections 2023-09-30T00:49:43.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.757+0000 7f255e57c700 1 -- 172.21.15.6:0/537161246 wait complete. 2023-09-30T00:49:43.770 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.901+0000 7f6ac134f700 10 monclient: get_monmap_and_config 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.901+0000 7f6ac134f700 10 monclient: build_initial_monmap 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.901+0000 7f6ac134f700 10 monclient: monmap: 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:43.901700+0000 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:43.901700+0000 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:43.901 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.902 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:43.902 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:43.902 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:43.902 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:43.902 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:43.902 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 Processor -- start 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 -- start start 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient: init 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient: _add_conns ranks=[1,2,3,4,0] 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f6abc098550 addr v1:172.21.15.22:6789/0 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 0x7f6abc1374e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): picked mon.noname-b con 0x7f6abc12f760 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6abc137a20 0x7f6abc139e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.906 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): picked mon.noname-a con 0x7f6abc137a20 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6abc096d60 con 0x7f6abc098550 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6abc12fbd0 con 0x7f6abc137a20 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc0a1c40 con 0x7f6abc12f760 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6ac134f700 10 monclient(hunting): authenticate will time out at 1580.199760s 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6aba7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 0x7f6abc1374e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.907 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6abc137a20 0x7f6abc139e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f6abc137a20 0x7f6abc139e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35880/0 (socket says 172.21.15.6:35880) 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6aba7fc700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 0x7f6abc1374e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40252/0 (socket says 172.21.15.6:40252) 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 1 -- 172.21.15.6:0/2696001356 learned_addr learned my addr 172.21.15.6:0/2696001356 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 10 monclient(hunting): get_auth_request con 0x7f6abc137a20 auth_method 0 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6abb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6aba7fc700 10 monclient(hunting): get_auth_request con 0x7f6abc12f760 auth_method 0 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6aba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6aba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.908 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.902+0000 7f6aba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6aba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6aba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6aba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 10 monclient(hunting): handle_auth_done global_id 4797 payload 274 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 1 -- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 msgr2=0x7f6abc1374e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:43.909 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 1 --2- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 0x7f6abc1374e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 1 -- 172.21.15.6:0/2696001356 >> v1:172.21.15.22:6789/0 conn(0x7f6abc098550 legacy=0x7f6abc098920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 10 monclient: _finish_hunting 0 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 1 monclient: found mon.noname-a 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6abb7fe700 1 -- 172.21.15.6:0/2696001356 --> v2:172.21.15.6:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc097040 con 0x7f6abc137a20 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.903+0000 7f6aba7fc700 1 --2- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 0x7f6abc1374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6abb7fe700 1 -- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc137a20 msgr2=0x7f6abc139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6abb7fe700 1 --2- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc137a20 0x7f6abc139e00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009f20 tx=0x7f6ab4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc5a2cca5700048d server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 1 -- 172.21.15.6:0/2696001356 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab400e070 con 0x7f6abc137a20 2023-09-30T00:49:43.910 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: dump: 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.911 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: _finish_auth 0 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 1 -- 172.21.15.6:0/2696001356 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ab400bba0 con 0x7f6abc137a20 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ac134f700 5 monclient: authenticate success, global_id 4797 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ac134f700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ac134f700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 1 -- 172.21.15.6:0/2696001356 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ab40092e0 con 0x7f6abc137a20 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ac134f700 4 set_mon_vals no callback set 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.905+0000 7f6ab9ffb700 10 monclient: dump: 2023-09-30T00:49:43.912 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 10 monclient: shutdown 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:43.913 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 -- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc137a20 msgr2=0x7f6abc139e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 --2- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc137a20 0x7f6abc139e00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009f20 tx=0x7f6ab4009840 comp rx=0 tx=0).stop 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 -- 172.21.15.6:0/2696001356 shutdown_connections 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 --2- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc137a20 0x7f6abc139e00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 --2- 172.21.15.6:0/2696001356 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc12f760 0x7f6abc1374e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 -- 172.21.15.6:0/2696001356 >> 172.21.15.6:0/2696001356 conn(0x7f6abc0a5480 msgr2=0x7f6abc0a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 -- 172.21.15.6:0/2696001356 shutdown_connections 2023-09-30T00:49:43.914 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.906+0000 7f6ac134f700 1 -- 172.21.15.6:0/2696001356 wait complete. 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.907+0000 7f6ac134f700 10 monclient: build_initial_monmap 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.907+0000 7f6ac134f700 10 monclient: monmap: 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:43.919 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.907+0000 7f6ac134f700 1 Processor -- start 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.907+0000 7f6ac134f700 1 -- start start 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.907+0000 7f6ac134f700 10 monclient: init 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient: _add_conns ranks=[2,0,1] 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 0x7f6abc135c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): picked mon.noname-c con 0x7f6abc098550 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 0x7f6abc09ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): picked mon.noname-a con 0x7f6abc12f760 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:43.920 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6abc137a20 0x7f6abc09b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): picked mon.noname-b con 0x7f6abc137a20 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc137620 con 0x7f6abc12f760 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc09c000 con 0x7f6abc137a20 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc135320 con 0x7f6abc098550 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6ac134f700 10 monclient(hunting): authenticate will time out at 1580.205701s 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 0x7f6abc135c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 0x7f6abc09ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.921 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 0x7f6abc09ab70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35882/0 (socket says 172.21.15.6:35882) 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 1 -- 172.21.15.6:0/1562113619 learned_addr learned my addr 172.21.15.6:0/1562113619 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 0x7f6abc135c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40264/0 (socket says 172.21.15.6:40264) 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): get_auth_request con 0x7f6abc12f760 auth_method 0 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): get_auth_request con 0x7f6abc098550 auth_method 0 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.922 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6aba7fc700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.908+0000 7f6abaffd700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abb7fe700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6abc137a20 0x7f6abc09b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abb7fe700 10 monclient(hunting): get_auth_request con 0x7f6abc137a20 auth_method 0 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abb7fe700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abb7fe700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abb7fe700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 10 monclient(hunting): handle_auth_done global_id 4853 payload 824 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 1 -- 172.21.15.6:0/1562113619 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6abc137a20 msgr2=0x7f6abc09b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.923 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6abc137a20 0x7f6abc09b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 1 -- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 msgr2=0x7f6abc09ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 0x7f6abc09ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 10 monclient: _finish_hunting 0 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 1 monclient: found mon.noname-c 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6abaffd700 1 -- 172.21.15.6:0/1562113619 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0009420 con 0x7f6abc098550 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.909+0000 7f6aba7fc700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 0x7f6abc09ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6abaffd700 1 -- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 msgr2=0x7f6abc135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6abaffd700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 0x7f6abc135c70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00c4e0 tx=0x7f6aac007ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac0117a0 con 0x7f6abc098550 2023-09-30T00:49:43.924 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: dump: 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.925 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: _finish_auth 0 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 5 monclient: authenticate success, global_id 4853 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6aac011920 con 0x7f6abc098550 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 10 monclient: _renew_subs 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac011ce0 con 0x7f6abc098550 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab4009ae0 con 0x7f6abc098550 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:43.926 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6aa2ffd700 10 monclient: dump: 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ab97fa700 4 set_mon_vals no callback set 2023-09-30T00:49:43.927 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 10 monclient: _renew_subs 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.911+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc133410 con 0x7f6abc098550 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f6aac012630 con 0x7f6abc098550 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6aa2ffd700 1 --2- 172.21.15.6:0/1562113619 >> v2:172.21.15.22:6812/142603 conn(0x7f6aa80357a0 0x7f6aa8037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f6aac0475e0 con 0x7f6abc098550 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6ac134f700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6ac134f700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6ac134f700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.912+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc0a2620 con 0x7f6abc098550 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.913+0000 7f6aba7fc700 1 --2- 172.21.15.6:0/1562113619 >> v2:172.21.15.22:6812/142603 conn(0x7f6aa80357a0 0x7f6aa8037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.913+0000 7f6aba7fc700 10 monclient: get_auth_request con 0x7f6aa80357a0 auth_method 0 2023-09-30T00:49:43.928 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.915+0000 7f6aba7fc700 1 --2- 172.21.15.6:0/1562113619 >> v2:172.21.15.22:6812/142603 conn(0x7f6aa80357a0 0x7f6aa8037c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0a0200 tx=0x7f6ab00091a0 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.917+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f6aac002090 con 0x7f6abc098550 2023-09-30T00:49:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.917+0000 7f6aa2ffd700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:43.929 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:43.917+0000 7f6aa2ffd700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:44.119 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.119+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f6abc135460 con 0x7f6aa80357a0 2023-09-30T00:49:44.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.120+0000 7f6aa2ffd700 1 -- 172.21.15.6:0/1562113619 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f6ab0014440 con 0x7f6aa80357a0 2023-09-30T00:49:44.224 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 >> v2:172.21.15.22:6812/142603 conn(0x7f6aa80357a0 msgr2=0x7f6aa8037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 --2- 172.21.15.6:0/1562113619 >> v2:172.21.15.22:6812/142603 conn(0x7f6aa80357a0 0x7f6aa8037c50 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f6abc0a0200 tx=0x7f6ab00091a0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 10 monclient: shutdown 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 msgr2=0x7f6abc135c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 0x7f6abc135c70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00c4e0 tx=0x7f6aac007ea0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 shutdown_connections 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 --2- 172.21.15.6:0/1562113619 >> v2:172.21.15.22:6812/142603 conn(0x7f6aa80357a0 0x7f6aa8037c50 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f6abc137a20 0x7f6abc09b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.225 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f6abc098550 0x7f6abc135c70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 --2- 172.21.15.6:0/1562113619 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f6abc12f760 0x7f6abc09ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 >> 172.21.15.6:0/1562113619 conn(0x7f6abc0a5480 msgr2=0x7f6abc132870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:44.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 shutdown_connections 2023-09-30T00:49:44.226 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.122+0000 7f6ac134f700 1 -- 172.21.15.6:0/1562113619 wait complete. 2023-09-30T00:49:44.226 INFO:teuthology.orchestra.run.smithi006.stderr:instructed osd(s) 5 to deep-scrub 2023-09-30T00:49:44.227 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:49:44.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.361+0000 7f9c8b7f7700 10 monclient: get_monmap_and_config 2023-09-30T00:49:44.362 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.361+0000 7f9c8b7f7700 10 monclient: build_initial_monmap 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.361+0000 7f9c8b7f7700 10 monclient: monmap: 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:49:44.363101+0000 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:49:44.363101+0000 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:49:44.363 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:49:44.364 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:44.364 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.362+0000 7f9c8b7f7700 1 Processor -- start 2023-09-30T00:49:44.367 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.362+0000 7f9c8b7f7700 1 -- start start 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.362+0000 7f9c8b7f7700 10 monclient: init 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient: _add_conns ranks=[4,3,2,0,1] 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 1 --2- >> v2:172.21.15.22:3300/0 conn(0x7f9c84098670 0x7f9c84132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): picked mon.noname-c con 0x7f9c84098670 addr v2:172.21.15.22:3300/0 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 0x7f9c8412ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): picked mon.noname-a con 0x7f9c84098a40 addr v2:172.21.15.6:3300/0 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8412f320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): picked mon.noname-b con 0x7f9c84098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c840a0a50 con 0x7f9c84098a40 2023-09-30T00:49:44.368 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 1 -- --> v2:172.21.15.22:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c840a1000 con 0x7f9c84098670 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840a0730 con 0x7f9c84098e90 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c8b7f7700 10 monclient(hunting): authenticate will time out at 1580.661311s 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8412f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 0x7f9c8412ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8412f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40270/0 (socket says 172.21.15.6:40270) 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 0x7f9c8412ede0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35884/0 (socket says 172.21.15.6:35884) 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 -- 172.21.15.6:0/3246037859 learned_addr learned my addr 172.21.15.6:0/3246037859 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): get_auth_request con 0x7f9c84098e90 auth_method 0 2023-09-30T00:49:44.369 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): get_auth_request con 0x7f9c84098a40 auth_method 0 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 1 --2- 172.21.15.6:0/3246037859 >> v2:172.21.15.22:3300/0 conn(0x7f9c84098670 0x7f9c84132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): get_auth_request con 0x7f9c84098670 auth_method 0 2023-09-30T00:49:44.370 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89593700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient(hunting): handle_auth_done global_id 4856 payload 274 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 -- 172.21.15.6:0/3246037859 >> v2:172.21.15.22:3300/0 conn(0x7f9c84098670 msgr2=0x7f9c84132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:44.371 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 --2- 172.21.15.6:0/3246037859 >> v2:172.21.15.22:3300/0 conn(0x7f9c84098670 0x7f9c84132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 -- 172.21.15.6:0/3246037859 >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 msgr2=0x7f9c8412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 --2- 172.21.15.6:0/3246037859 >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 0x7f9c8412ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient: _finish_hunting 0 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 monclient: found mon.noname-b 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c89d94700 1 -- 172.21.15.6:0/3246037859 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c84068df0 con 0x7f9c84098e90 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.363+0000 7f9c88d92700 1 --2- 172.21.15.6:0/3246037859 >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 0x7f9c8412ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c89d94700 1 -- 172.21.15.6:0/3246037859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 msgr2=0x7f9c8412f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c89d94700 1 --2- 172.21.15.6:0/3246037859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8412f320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9c8409fe60 tx=0x7f9c7c009840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3096edd0475324e9 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 1 -- 172.21.15.6:0/3246037859 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c00e070 con 0x7f9c84098e90 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:44.372 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: dump: 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:44.373 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: _finish_auth 0 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 1 -- 172.21.15.6:0/3246037859 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c00bb80 con 0x7f9c84098e90 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 5 monclient: authenticate success, global_id 4856 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 1 -- 172.21.15.6:0/3246037859 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c0092e0 con 0x7f9c84098e90 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 10 monclient: get_monmap_and_config success 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 4 set_mon_vals no callback set 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c77fff700 10 monclient: dump: 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:44.374 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 10 monclient: shutdown 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/3246037859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 msgr2=0x7f9c8412f320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.375 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/3246037859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8412f320 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f9c8409fe60 tx=0x7f9c7c009840 comp rx=0 tx=0).stop 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/3246037859 shutdown_connections 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/3246037859 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8412f320 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/3246037859 >> v2:172.21.15.22:3300/0 conn(0x7f9c84098670 0x7f9c84132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/3246037859 >> v2:172.21.15.6:3300/0 conn(0x7f9c84098a40 0x7f9c8412ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.366+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/3246037859 >> 172.21.15.6:0/3246037859 conn(0x7f9c840a5480 msgr2=0x7f9c840a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.367+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/3246037859 shutdown_connections 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.367+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/3246037859 wait complete. 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.367+0000 7f9c8b7f7700 10 monclient: build_initial_monmap 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.367+0000 7f9c8b7f7700 10 monclient: monmap: 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:49:44.376 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:49:44.377 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:44.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.367+0000 7f9c8b7f7700 1 Processor -- start 2023-09-30T00:49:44.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 -- start start 2023-09-30T00:49:44.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient: init 2023-09-30T00:49:44.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient: _reopen_session rank -1 2023-09-30T00:49:44.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient: _add_conns ranks=[1,0,2] 2023-09-30T00:49:44.381 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c84098670 0x7f9c8412be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): picked mon.noname-b con 0x7f9c84098670 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 0x7f9c8412c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): picked mon.noname-a con 0x7f9c84098a40 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8409c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): picked mon.noname-c con 0x7f9c84098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840a1500 con 0x7f9c84098a40 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840a3910 con 0x7f9c84098670 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): start opening mon connection 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c840997c0 con 0x7f9c84098e90 2023-09-30T00:49:44.382 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): _renew_subs 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c8b7f7700 10 monclient(hunting): authenticate will time out at 1580.666747s 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8409c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 0x7f9c8412c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8409c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:40274/0 (socket says 172.21.15.6:40274) 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 0x7f9c8412c3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:35890/0 (socket says 172.21.15.6:35890) 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 1 -- 172.21.15.6:0/2137416887 learned_addr learned my addr 172.21.15.6:0/2137416887 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): get_auth_request con 0x7f9c84098e90 auth_method 0 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 10 monclient(hunting): get_auth_request con 0x7f9c84098a40 auth_method 0 2023-09-30T00:49:44.383 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89d94700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c88d92700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.368+0000 7f9c89593700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c84098670 0x7f9c8412be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c88d92700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89593700 10 monclient(hunting): get_auth_request con 0x7f9c84098670 auth_method 0 2023-09-30T00:49:44.384 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89593700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89593700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89593700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 10 monclient(hunting): handle_auth_done global_id 4859 payload 824 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 1 -- 172.21.15.6:0/2137416887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c84098670 msgr2=0x7f9c8412be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c84098670 0x7f9c8412be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 1 -- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 msgr2=0x7f9c8412c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 0x7f9c8412c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 10 monclient: _finish_hunting 0 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 1 monclient: found mon.noname-c 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:49:44.385 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c89d94700 1 -- 172.21.15.6:0/2137416887 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c78009420 con 0x7f9c84098e90 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.369+0000 7f9c88d92700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 0x7f9c8412c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c89d94700 1 -- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 msgr2=0x7f9c8409c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c89d94700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8409c7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9c840a0870 tx=0x7f9c7c018000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c007a30 con 0x7f9c84098e90 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: dump: 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:44.386 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: _finish_auth 0 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 5 monclient: authenticate success, global_id 4859 2023-09-30T00:49:44.387 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7c007b90 con 0x7f9c84098e90 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 10 monclient: _renew_subs 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c009ae0 con 0x7f9c84098e90 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c026460 con 0x7f9c84098e90 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c757fa700 10 monclient: dump: 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:49:44.388 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c777fe700 4 set_mon_vals no callback set 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 10 monclient: _renew_subs 2023-09-30T00:49:44.389 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.371+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c8409df60 con 0x7f9c84098e90 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.372+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f9c7c026850 con 0x7f9c84098e90 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.372+0000 7f9c8b7f7700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.372+0000 7f9c8b7f7700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.372+0000 7f9c8b7f7700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.372+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c84072b00 con 0x7f9c84098e90 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.373+0000 7f9c757fa700 1 --2- 172.21.15.6:0/2137416887 >> v2:172.21.15.22:6812/142603 conn(0x7f9c680357a0 0x7f9c68037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.373+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9c7c049600 con 0x7f9c84098e90 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.373+0000 7f9c89593700 1 --2- 172.21.15.6:0/2137416887 >> v2:172.21.15.22:6812/142603 conn(0x7f9c680357a0 0x7f9c68037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:49:44.390 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.373+0000 7f9c89593700 10 monclient: get_auth_request con 0x7f9c680357a0 auth_method 0 2023-09-30T00:49:44.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.375+0000 7f9c89593700 1 --2- 172.21.15.6:0/2137416887 >> v2:172.21.15.22:6812/142603 conn(0x7f9c680357a0 0x7f9c68037c50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009d40 tx=0x7f9c80008040 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:49:44.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.378+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f9c7c01c770 con 0x7f9c84098e90 2023-09-30T00:49:44.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.378+0000 7f9c757fa700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:49:44.391 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.378+0000 7f9c757fa700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:49:44.578 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.577+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f9c84099900 con 0x7f9c680357a0 2023-09-30T00:49:44.580 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.580+0000 7f9c757fa700 1 -- 172.21.15.6:0/2137416887 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53734 (secure 0 0 0) 0x7f9c84099900 con 0x7f9c680357a0 2023-09-30T00:49:44.581 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:49:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 >> v2:172.21.15.22:6812/142603 conn(0x7f9c680357a0 msgr2=0x7f9c68037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/2137416887 >> v2:172.21.15.22:6812/142603 conn(0x7f9c680357a0 0x7f9c68037c50 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009d40 tx=0x7f9c80008040 comp rx=0 tx=0).stop 2023-09-30T00:49:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 10 monclient: shutdown 2023-09-30T00:49:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:49:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 msgr2=0x7f9c8409c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:49:44.583 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8409c7a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f9c840a0870 tx=0x7f9c7c018000 comp rx=0 tx=0).stop 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 shutdown_connections 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/2137416887 >> v2:172.21.15.22:6812/142603 conn(0x7f9c680357a0 0x7f9c68037c50 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9c84098e90 0x7f9c8409c7a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9c84098670 0x7f9c8412be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 --2- 172.21.15.6:0/2137416887 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9c84098a40 0x7f9c8412c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 >> 172.21.15.6:0/2137416887 conn(0x7f9c840a5480 msgr2=0x7f9c84069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 shutdown_connections 2023-09-30T00:49:44.584 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:49:44.582+0000 7f9c8b7f7700 1 -- 172.21.15.6:0/2137416887 wait complete. 2023-09-30T00:49:44.585 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:49:44.596 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":172,"stamp":"2023-09-30T00:49:43.607356+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31880,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566199160,"statfs":{"total":579820584960,"available":579787939840,"internally_reserved":0,"allocated":1007616,"data_stored":1327094,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002514"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.431391+0000","last_change":"2023-09-30T00:44:28.427091+0000","last_active":"2023-09-30T00:44:28.431391+0000","last_peered":"2023-09-30T00:44:28.431391+0000","last_clean":"2023-09-30T00:44:28.431391+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:44:28.431391+0000","last_undegraded":"2023-09-30T00:44:28.431391+0000","last_fullsized":"2023-09-30T00:44:28.431391+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T05:47:25.053141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.427410+0000","last_change":"2023-09-30T00:44:28.426953+0000","last_active":"2023-09-30T00:44:28.427410+0000","last_peered":"2023-09-30T00:44:28.427410+0000","last_clean":"2023-09-30T00:44:28.427410+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:44:28.427410+0000","last_undegraded":"2023-09-30T00:44:28.427410+0000","last_fullsized":"2023-09-30T00:44:28.427410+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T12:21:01.653238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426484+0000","last_change":"2023-09-30T00:44:28.423208+0000","last_active":"2023-09-30T00:44:28.426484+0000","last_peered":"2023-09-30T00:44:28.426484+0000","last_clean":"2023-09-30T00:44:28.426484+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:44:28.426484+0000","last_undegraded":"2023-09-30T00:44:28.426484+0000","last_fullsized":"2023-09-30T00:44:28.426484+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T10:04:55.556087+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426680+0000","last_change":"2023-09-30T00:44:28.423004+0000","last_active":"2023-09-30T00:44:28.426680+0000","last_peered":"2023-09-30T00:44:28.426680+0000","last_clean":"2023-09-30T00:44:28.426680+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:44:28.426680+0000","last_undegraded":"2023-09-30T00:44:28.426680+0000","last_fullsized":"2023-09-30T00:44:28.426680+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T04:22:00.541043+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"14'117","reported_seq":162,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.417426+0000","last_change":"2023-09-30T00:44:25.575831+0000","last_active":"2023-09-30T00:44:28.417426+0000","last_peered":"2023-09-30T00:44:28.417426+0000","last_clean":"2023-09-30T00:44:28.417426+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:44:28.417426+0000","last_undegraded":"2023-09-30T00:44:28.417426+0000","last_fullsized":"2023-09-30T00:44:28.417426+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:24.231195+0000","last_clean_scrub_stamp":"2023-09-30T00:44:24.231195+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:38:05.898656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424226+0000","last_change":"2023-09-30T00:44:28.422311+0000","last_active":"2023-09-30T00:44:28.424226+0000","last_peered":"2023-09-30T00:44:28.424226+0000","last_clean":"2023-09-30T00:44:28.424226+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:44:28.424226+0000","last_undegraded":"2023-09-30T00:44:28.424226+0000","last_fullsized":"2023-09-30T00:44:28.424226+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:37:58.120817+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.423390+0000","last_change":"2023-09-30T00:44:28.422342+0000","last_active":"2023-09-30T00:44:28.423390+0000","last_peered":"2023-09-30T00:44:28.423390+0000","last_clean":"2023-09-30T00:44:28.423390+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:44:28.423390+0000","last_undegraded":"2023-09-30T00:44:28.423390+0000","last_fullsized":"2023-09-30T00:44:28.423390+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:53:26.637840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.426632+0000","last_change":"2023-09-30T00:44:28.424009+0000","last_active":"2023-09-30T00:44:28.426632+0000","last_peered":"2023-09-30T00:44:28.426632+0000","last_clean":"2023-09-30T00:44:28.426632+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:44:28.426632+0000","last_undegraded":"2023-09-30T00:44:28.426632+0000","last_fullsized":"2023-09-30T00:44:28.426632+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:38:57.243406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":9,"reported_epoch":16,"state":"active+clean","last_fresh":"2023-09-30T00:44:28.424888+0000","last_change":"2023-09-30T00:44:28.424000+0000","last_active":"2023-09-30T00:44:28.424888+0000","last_peered":"2023-09-30T00:44:28.424888+0000","last_clean":"2023-09-30T00:44:28.424888+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:44:28.424888+0000","last_undegraded":"2023-09-30T00:44:28.424888+0000","last_fullsized":"2023-09-30T00:44:28.424888+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:44:27.404013+0000","last_clean_scrub_stamp":"2023-09-30T00:44:27.404013+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T09:57:56.916213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.42999999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.432}]},{"osd":1,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41399999999999998}]},{"osd":2,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.63300000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.496}]},{"osd":3,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.317},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.54700000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36899999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.32600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.48099999999999998}]}]},{"osd":4,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73199999999999998,"5min":2.044,"15min":2.044},"last":0.58499999999999996}]},{"osd":1,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.64200000000000002},"min":{"1min":0.46300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77900000000000003,"5min":1.833,"15min":1.833},"last":0.63800000000000001}]},{"osd":2,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67500000000000004,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.54100000000000004}]},{"osd":3,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41199999999999998,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51500000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.628,"5min":0.746,"15min":0.746},"last":0.60099999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47399999999999998,"15min":0.56799999999999995},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.64200000000000002,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.49199999999999999,"15min":0.59199999999999997},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64900000000000002,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":0.65600000000000003}]}]},{"osd":3,"up_from":11,"seq":47244640321,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65300000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63800000000000001}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.505}]},{"osd":2,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.70699999999999996}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.42299999999999999},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":0.876,"15min":0.876},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45700000000000002,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.65400000000000003,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.624}]}]},{"osd":2,"up_from":11,"seq":47244640322,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57999999999999996,"5min":1.298,"15min":1.298},"last":0.42899999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56299999999999994,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.39000000000000001,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.5,"15min":1.5},"last":0.27700000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73699999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":1.343,"15min":1.343},"last":0.47099999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71199999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.503},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.746,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.41499999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78200000000000003,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77500000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.44400000000000001}]}]},{"osd":0,"up_from":11,"seq":47244640321,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.35399999999999998,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.505,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.29699999999999999}]},{"osd":2,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.377,"15min":0.372},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.34000000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.435,"5min":0.37,"15min":0.37},"max":{"1min":0.65300000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.59299999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.627,"5min":0.623,"15min":0.629},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.621}]},{"osd":5,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.57099999999999995}]}]},{"osd":1,"up_from":11,"seq":47244640321,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34200000000000003},"min":{"1min":0.28100000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.433},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.37,"15min":0.372},"min":{"1min":0.30099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.52400000000000002,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.40200000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.38700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.503},{"interface":"front","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.47599999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54100000000000004,"15min":0.55400000000000005},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79000000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77000000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.53800000000000003}]},{"osd":4,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77300000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.59499999999999997},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78200000000000003,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.60499999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.745,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.78000000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.629}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:49:44.600 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.600 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.600 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2023-09-30T00:44:24.231195+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=24, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2023-09-30T00:44:27.404013+0000 time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=44, tm_sec=27, tm_wday=5, tm_yday=273, tm_isdst=-1) <= time.struct_time(tm_year=2023, tm_mon=9, tm_mday=30, tm_hour=0, tm_min=49, tm_sec=39, tm_wday=5, tm_yday=273, tm_isdst=0) 2023-09-30T00:49:44.601 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2023-09-30T00:49:59.999 INFO:tasks.ceph.mon.a.smithi006.stderr:2023-09-30T00:49:59.999+0000 7f82600df700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR Module 'pg_autoscaler' has failed: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:50:04.603 DEBUG:teuthology.orchestra.run.smithi006:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2023-09-30T00:50:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.744+0000 7f913cc32700 10 monclient: get_monmap_and_config 2023-09-30T00:50:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.744+0000 7f913cc32700 10 monclient: build_initial_monmap 2023-09-30T00:50:04.745 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.745+0000 7f913cc32700 10 monclient: monmap: 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:50:04.746214+0000 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:50:04.746214+0000 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:50:04.746 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.745+0000 7f913cc32700 1 Processor -- start 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.745+0000 7f913cc32700 1 -- start start 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.745+0000 7f913cc32700 10 monclient: init 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient: _add_conns ranks=[0,2,1,3,4] 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient(hunting): picked mon.noname-a-legacy con 0x7f9138098670 addr v1:172.21.15.6:6789/0 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 0x7f9138137620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient(hunting): picked mon.noname-b con 0x7f9138132d00 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7f9138137b60 addr v1:172.21.15.22:6789/0 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 1 -- --> v1:172.21.15.6:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91380a1000 con 0x7f9138098670 2023-09-30T00:50:04.749 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91380a0730 con 0x7f9138137b60 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9138099150 con 0x7f9138132d00 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913cc32700 10 monclient(hunting): authenticate will time out at 1601.044419s 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 0x7f9138137620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913659c700 1 --1- >> v1:172.21.15.6:6789/0 conn(0x7f9138098670 0x7f9138098a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.6:6789/0 says I am v1:172.21.15.6:40116/0 (socket says 172.21.15.6:40116) 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f913659c700 1 -- 172.21.15.6:0/996411069 learned_addr learned my addr 172.21.15.6:0/996411069 (peer_addr_for_me v1:172.21.15.6:0/0) 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): get_auth_request con 0x7f9138132d00 auth_method 0 2023-09-30T00:50:04.750 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient(hunting): handle_auth_done global_id 4862 payload 274 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 1 -- 172.21.15.6:0/996411069 >> v1:172.21.15.22:6789/0 conn(0x7f9138137b60 legacy=0x7f9138139f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 1 -- 172.21.15.6:0/996411069 >> v1:172.21.15.6:6789/0 conn(0x7f9138098670 legacy=0x7f9138098a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient: _finish_hunting 0 2023-09-30T00:50:04.751 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 1 monclient: found mon.noname-b 2023-09-30T00:50:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:50:04.752 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.746+0000 7f9135d9b700 1 -- 172.21.15.6:0/996411069 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9138068df0 con 0x7f9138132d00 2023-09-30T00:50:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f9135d9b700 1 -- 172.21.15.6:0/996411069 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 msgr2=0x7f9138137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:50:04.757 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f9135d9b700 1 --2- 172.21.15.6:0/996411069 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 0x7f9138137620 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9120009910 tx=0x7f91200091a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1836c9f4092e9705 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 1 -- 172.21.15.6:0/996411069 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f912000b9f0 con 0x7f9138132d00 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: dump: 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:04.758 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: _finish_auth 0 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 1 -- 172.21.15.6:0/996411069 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f912000bb50 con 0x7f9138132d00 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 5 monclient: authenticate success, global_id 4862 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 20 monclient: get_monmap_and_config waiting for monmap|config 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 1 -- 172.21.15.6:0/996411069 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120011420 con 0x7f9138132d00 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 4 set_mon_vals no callback set 2023-09-30T00:50:04.759 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913559a700 10 monclient: dump: 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:04.760 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 10 monclient: shutdown 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 1 -- 172.21.15.6:0/996411069 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 msgr2=0x7f9138137620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 1 --2- 172.21.15.6:0/996411069 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 0x7f9138137620 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9120009910 tx=0x7f91200091a0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 1 -- 172.21.15.6:0/996411069 shutdown_connections 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 1 --2- 172.21.15.6:0/996411069 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138132d00 0x7f9138137620 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.749+0000 7f913cc32700 1 -- 172.21.15.6:0/996411069 >> 172.21.15.6:0/996411069 conn(0x7f91380a5480 msgr2=0x7f91380a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.750+0000 7f913cc32700 1 -- 172.21.15.6:0/996411069 shutdown_connections 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.750+0000 7f913cc32700 1 -- 172.21.15.6:0/996411069 wait complete. 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.750+0000 7f913cc32700 10 monclient: build_initial_monmap 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.750+0000 7f913cc32700 10 monclient: monmap: 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:50:04.761 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.751+0000 7f913cc32700 1 Processor -- start 2023-09-30T00:50:04.762 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.751+0000 7f913cc32700 1 -- start start 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.751+0000 7f913cc32700 10 monclient: init 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient: _add_conns ranks=[2,1,0] 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 0x7f913809efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): picked mon.noname-c con 0x7f9138098670 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9138132d00 0x7f9138133db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): picked mon.noname-b con 0x7f9138132d00 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9138137b60 0x7f91381342f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): picked mon.noname-a con 0x7f9138137b60 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.763 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913812ed90 con 0x7f9138137b60 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91380a1500 con 0x7f9138132d00 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91380a33b0 con 0x7f9138098670 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913cc32700 10 monclient(hunting): authenticate will time out at 1601.050490s 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 0x7f913809efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 0x7f913809efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:54758/0 (socket says 172.21.15.6:54758) 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 -- 172.21.15.6:0/54264966 learned_addr learned my addr 172.21.15.6:0/54264966 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:50:04.764 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): get_auth_request con 0x7f9138098670 auth_method 0 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9138137b60 0x7f91381342f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): get_auth_request con 0x7f9138137b60 auth_method 0 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9138132d00 0x7f9138133db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.765 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): get_auth_request con 0x7f9138132d00 auth_method 0 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9136d9d700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.766 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f9135d9b700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient(hunting): handle_auth_done global_id 4865 payload 824 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 -- 172.21.15.6:0/54264966 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9138132d00 msgr2=0x7f9138133db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9138132d00 0x7f9138133db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 -- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9138137b60 msgr2=0x7f91381342f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9138137b60 0x7f91381342f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient: _finish_hunting 0 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 monclient: found mon.noname-c 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:04.767 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 10 monclient: _send_mon_message to mon.noname-c at v2:172.21.15.6:3301/0 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.752+0000 7f913659c700 1 -- 172.21.15.6:0/54264966 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9120009420 con 0x7f9138098670 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913659c700 1 -- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 msgr2=0x7f913809efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913659c700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 0x7f913809efb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f913000c5e0 tx=0x7f9130004a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130005d40 con 0x7f9138098670 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: dump: 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:04.768 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: _finish_auth 0 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91300070d0 con 0x7f9138098670 2023-09-30T00:50:04.769 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 5 monclient: authenticate success, global_id 4865 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9130007490 con 0x7f9138098670 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 10 monclient: _renew_subs 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91381336b0 con 0x7f9138098670 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f9134d99700 4 set_mon_vals no callback set 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f912e7fc700 10 monclient: dump: 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:04.770 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 10 monclient: _renew_subs 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:50:04.771 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.755+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91381370c0 con 0x7f9138098670 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mon.2 v2:172.21.15.6:3301/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7f913001b3b0 con 0x7f9138098670 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f913cc32700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f913cc32700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f913cc32700 10 monclient: _send_mon_message to mon.c at v2:172.21.15.6:3301/0 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9138072b00 con 0x7f9138098670 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f912e7fc700 1 --2- 172.21.15.6:0/54264966 >> v2:172.21.15.22:6812/142603 conn(0x7f91180357a0 0x7f9118037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mon.2 v2:172.21.15.6:3301/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7f9130046ef0 con 0x7f9138098670 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f9135d9b700 1 --2- 172.21.15.6:0/54264966 >> v2:172.21.15.22:6812/142603 conn(0x7f91180357a0 0x7f9118037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:04.772 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.756+0000 7f9135d9b700 10 monclient: get_auth_request con 0x7f91180357a0 auth_method 0 2023-09-30T00:50:04.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.759+0000 7f9135d9b700 1 --2- 172.21.15.6:0/54264966 >> v2:172.21.15.22:6812/142603 conn(0x7f91180357a0 0x7f9118037c50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f91200030c0 tx=0x7f9120016000 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:04.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.761+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mon.2 v2:172.21.15.6:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7f913000f490 con 0x7f9138098670 2023-09-30T00:50:04.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.761+0000 7f912e7fc700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:04.773 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.761+0000 7f912e7fc700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:04.957 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.957+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 --> v2:172.21.15.22:6812/142603 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f913812fce0 con 0x7f91180357a0 2023-09-30T00:50:04.960 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.959+0000 7f912e7fc700 1 -- 172.21.15.6:0/54264966 <== mgr.4102 v2:172.21.15.22:6812/142603 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+53921 (secure 0 0 0) 0x7f913812fce0 con 0x7f91180357a0 2023-09-30T00:50:04.961 INFO:teuthology.orchestra.run.smithi006.stdout: 2023-09-30T00:50:04.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 >> v2:172.21.15.22:6812/142603 conn(0x7f91180357a0 msgr2=0x7f9118037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 --2- 172.21.15.6:0/54264966 >> v2:172.21.15.22:6812/142603 conn(0x7f91180357a0 0x7f9118037c50 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f91200030c0 tx=0x7f9120016000 comp rx=0 tx=0).stop 2023-09-30T00:50:04.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 10 monclient: shutdown 2023-09-30T00:50:04.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:04.962 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 msgr2=0x7f913809efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 0x7f913809efb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f913000c5e0 tx=0x7f9130004a80 comp rx=0 tx=0).stop 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 shutdown_connections 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 --2- 172.21.15.6:0/54264966 >> v2:172.21.15.22:6812/142603 conn(0x7f91180357a0 0x7f9118037c50 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7f9138137b60 0x7f91381342f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7f9138098670 0x7f913809efb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 --2- 172.21.15.6:0/54264966 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7f9138132d00 0x7f9138133db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 >> 172.21.15.6:0/54264966 conn(0x7f91380a5480 msgr2=0x7f9138131af0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 shutdown_connections 2023-09-30T00:50:04.963 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:04.961+0000 7f913cc32700 1 -- 172.21.15.6:0/54264966 wait complete. 2023-09-30T00:50:04.964 INFO:teuthology.orchestra.run.smithi006.stderr:dumped all 2023-09-30T00:50:04.975 INFO:teuthology.orchestra.run.smithi006.stdout:{"pg_ready":true,"pg_map":{"version":182,"stamp":"2023-09-30T00:50:03.611839+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":31880,"kb_used_data":984,"kb_used_omap":0,"kb_used_meta":30848,"kb_avail":566199160,"statfs":{"total":579820584960,"available":579787939840,"internally_reserved":0,"allocated":1007616,"data_stored":1327094,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":31588352},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002953"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:50.846291+0000","last_change":"2023-09-30T00:49:50.846263+0000","last_active":"2023-09-30T00:49:50.846291+0000","last_peered":"2023-09-30T00:49:50.846291+0000","last_clean":"2023-09-30T00:49:50.846291+0000","last_became_active":"2023-09-30T00:44:28.426755+0000","last_became_peered":"2023-09-30T00:44:28.426755+0000","last_unstale":"2023-09-30T00:49:50.846291+0000","last_undegraded":"2023-09-30T00:49:50.846291+0000","last_fullsized":"2023-09-30T00:49:50.846291+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:50.846144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:50.846144+0000","last_clean_scrub_stamp":"2023-09-30T00:49:50.846144+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:13:13.058818+0000","scrub_duration":0.0015815110000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:44.556447+0000","last_change":"2023-09-30T00:49:44.556414+0000","last_active":"2023-09-30T00:49:44.556447+0000","last_peered":"2023-09-30T00:49:44.556447+0000","last_clean":"2023-09-30T00:49:44.556447+0000","last_became_active":"2023-09-30T00:44:28.426717+0000","last_became_peered":"2023-09-30T00:44:28.426717+0000","last_unstale":"2023-09-30T00:49:44.556447+0000","last_undegraded":"2023-09-30T00:49:44.556447+0000","last_fullsized":"2023-09-30T00:49:44.556447+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:44.556313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:44.556313+0000","last_clean_scrub_stamp":"2023-09-30T00:49:44.556313+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T08:10:30.811029+0000","scrub_duration":0.0016903070000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:45.872713+0000","last_change":"2023-09-30T00:49:45.872685+0000","last_active":"2023-09-30T00:49:45.872713+0000","last_peered":"2023-09-30T00:49:45.872713+0000","last_clean":"2023-09-30T00:49:45.872713+0000","last_became_active":"2023-09-30T00:44:28.422907+0000","last_became_peered":"2023-09-30T00:44:28.422907+0000","last_unstale":"2023-09-30T00:49:45.872713+0000","last_undegraded":"2023-09-30T00:49:45.872713+0000","last_fullsized":"2023-09-30T00:49:45.872713+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:45.872582+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:45.872582+0000","last_clean_scrub_stamp":"2023-09-30T00:49:45.872582+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:07:02.237017+0000","scrub_duration":0.001944351,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:41.753437+0000","last_change":"2023-09-30T00:49:41.753403+0000","last_active":"2023-09-30T00:49:41.753437+0000","last_peered":"2023-09-30T00:49:41.753437+0000","last_clean":"2023-09-30T00:49:41.753437+0000","last_became_active":"2023-09-30T00:44:28.422398+0000","last_became_peered":"2023-09-30T00:44:28.422398+0000","last_unstale":"2023-09-30T00:49:41.753437+0000","last_undegraded":"2023-09-30T00:49:41.753437+0000","last_fullsized":"2023-09-30T00:49:41.753437+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:41.753253+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:41.753253+0000","last_clean_scrub_stamp":"2023-09-30T00:49:41.753253+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T03:15:30.163560+0000","scrub_duration":0.0022782760000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"17'119","reported_seq":175,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:41.912113+0000","last_change":"2023-09-30T00:49:41.912078+0000","last_active":"2023-09-30T00:49:41.912113+0000","last_peered":"2023-09-30T00:49:41.912113+0000","last_clean":"2023-09-30T00:49:41.912113+0000","last_became_active":"2023-09-30T00:44:25.574844+0000","last_became_peered":"2023-09-30T00:44:25.574844+0000","last_unstale":"2023-09-30T00:49:41.912113+0000","last_undegraded":"2023-09-30T00:49:41.912113+0000","last_fullsized":"2023-09-30T00:49:41.912113+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"17'119","last_scrub_stamp":"2023-09-30T00:49:41.911993+0000","last_deep_scrub":"17'119","last_deep_scrub_stamp":"2023-09-30T00:49:41.911993+0000","last_clean_scrub_stamp":"2023-09-30T00:49:41.911993+0000","objects_scrubbed":2,"log_size":119,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T07:33:15.426213+0000","scrub_duration":0.0045806500000000003,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:49.880053+0000","last_change":"2023-09-30T00:49:49.880023+0000","last_active":"2023-09-30T00:49:49.880053+0000","last_peered":"2023-09-30T00:49:49.880053+0000","last_clean":"2023-09-30T00:49:49.880053+0000","last_became_active":"2023-09-30T00:44:28.421915+0000","last_became_peered":"2023-09-30T00:44:28.421915+0000","last_unstale":"2023-09-30T00:49:49.880053+0000","last_undegraded":"2023-09-30T00:49:49.880053+0000","last_fullsized":"2023-09-30T00:49:49.880053+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:49.879918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:49.879918+0000","last_clean_scrub_stamp":"2023-09-30T00:49:49.879918+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T06:56:16.051461+0000","scrub_duration":0.00156412,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:46.692967+0000","last_change":"2023-09-30T00:49:46.692936+0000","last_active":"2023-09-30T00:49:46.692967+0000","last_peered":"2023-09-30T00:49:46.692967+0000","last_clean":"2023-09-30T00:49:46.692967+0000","last_became_active":"2023-09-30T00:44:28.422021+0000","last_became_peered":"2023-09-30T00:44:28.422021+0000","last_unstale":"2023-09-30T00:49:46.692967+0000","last_undegraded":"2023-09-30T00:49:46.692967+0000","last_fullsized":"2023-09-30T00:49:46.692967+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:46.692821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:46.692821+0000","last_clean_scrub_stamp":"2023-09-30T00:49:46.692821+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T02:27:19.335028+0000","scrub_duration":0.0016127610000000001,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:42.483206+0000","last_change":"2023-09-30T00:49:42.483176+0000","last_active":"2023-09-30T00:49:42.483206+0000","last_peered":"2023-09-30T00:49:42.483206+0000","last_clean":"2023-09-30T00:49:42.483206+0000","last_became_active":"2023-09-30T00:44:28.423721+0000","last_became_peered":"2023-09-30T00:44:28.423721+0000","last_unstale":"2023-09-30T00:49:42.483206+0000","last_undegraded":"2023-09-30T00:49:42.483206+0000","last_fullsized":"2023-09-30T00:49:42.483206+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:42.483081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:42.483081+0000","last_clean_scrub_stamp":"2023-09-30T00:49:42.483081+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:20:01.257626+0000","scrub_duration":0.0090719019999999997,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":17,"state":"active+clean","last_fresh":"2023-09-30T00:49:45.230446+0000","last_change":"2023-09-30T00:49:45.230416+0000","last_active":"2023-09-30T00:49:45.230446+0000","last_peered":"2023-09-30T00:49:45.230446+0000","last_clean":"2023-09-30T00:49:45.230446+0000","last_became_active":"2023-09-30T00:44:28.423681+0000","last_became_peered":"2023-09-30T00:44:28.423681+0000","last_unstale":"2023-09-30T00:49:45.230446+0000","last_undegraded":"2023-09-30T00:49:45.230446+0000","last_fullsized":"2023-09-30T00:49:45.230446+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-09-30T00:49:45.230294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-09-30T00:49:45.230294+0000","last_clean_scrub_stamp":"2023-09-30T00:49:45.230294+0000","objects_scrubbed":0,"log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2023-10-01T01:32:54.581629+0000","scrub_duration":0.0015075819999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":59166,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":11,"seq":47244640325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.58199999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67600000000000005,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56899999999999995,"15min":0.55600000000000005},"min":{"1min":0.42999999999999999,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.67600000000000005,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.378}]},{"osd":1,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.52200000000000002,"15min":0.503},"min":{"1min":0.31,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.67100000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53400000000000003,"15min":0.54000000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.45700000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53900000000000003,"15min":0.52300000000000002},"min":{"1min":0.41599999999999998,"5min":0.379,"15min":0.379},"max":{"1min":0.69599999999999995,"5min":0.75,"15min":0.75},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.53800000000000003,"15min":0.53400000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.63300000000000001,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.47299999999999998}]},{"osd":3,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.42099999999999999,"15min":0.40000000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.64700000000000002,"5min":0.747,"15min":0.747},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.42699999999999999,"15min":0.40899999999999997},"min":{"1min":0.28899999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.54700000000000004,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.48999999999999999}]},{"osd":4,"last update":"Sat Sep 30 00:49:34 2023","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.47299999999999998,"15min":0.46300000000000002},"min":{"1min":0.32600000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67100000000000004,"5min":0.68400000000000005,"15min":0.68400000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.313,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":0.68300000000000005,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.56000000000000005}]}]},{"osd":4,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.51100000000000001,"15min":0.51700000000000002},"min":{"1min":0.33800000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.72699999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.54300000000000004,"15min":0.59199999999999997},"min":{"1min":0.40999999999999998,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.73199999999999998,"5min":2.044,"15min":2.044},"last":0.52600000000000002}]},{"osd":1,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55900000000000005,"15min":0.57899999999999996},"min":{"1min":0.41999999999999998,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58799999999999997,"15min":0.64200000000000002},"min":{"1min":0.46300000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.77900000000000003,"5min":1.833,"15min":1.833},"last":0.625}]},{"osd":2,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.53500000000000003,"15min":0.58499999999999996},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.67500000000000004,"5min":1.8180000000000001,"15min":1.8180000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.52800000000000002,"15min":0.53200000000000003},"min":{"1min":0.36199999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.63600000000000001}]},{"osd":3,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.41199999999999998,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.51500000000000001,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48299999999999998,"15min":0.47999999999999998},"min":{"1min":0.32900000000000001,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.628,"5min":0.746,"15min":0.746},"last":0.47399999999999998}]},{"osd":5,"last update":"Sat Sep 30 00:49:30 2023","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.47399999999999998,"15min":0.56799999999999995},"min":{"1min":0.314,"5min":0.311,"15min":0.311},"max":{"1min":0.64200000000000002,"5min":4.2889999999999997,"15min":4.2889999999999997},"last":0.434},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.49199999999999999,"15min":0.59199999999999997},"min":{"1min":0.29599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.64900000000000002,"5min":4.0170000000000003,"15min":4.0170000000000003},"last":0.375}]}]},{"osd":3,"up_from":11,"seq":47244640325,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55500000000000005,"15min":0.58199999999999996},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.65300000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.56499999999999995,"15min":0.57099999999999995},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.65200000000000002,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.60999999999999999}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53600000000000003,"15min":0.54200000000000004},"min":{"1min":0.42499999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73799999999999999,"5min":0.73799999999999999,"15min":0.73799999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.497,"5min":0.51800000000000002,"15min":0.53600000000000003},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64600000000000002,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.53300000000000003}]},{"osd":2,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54400000000000004,"15min":0.55100000000000005},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.75800000000000001,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.501},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.54900000000000004,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68899999999999995,"5min":1.1299999999999999,"15min":1.1299999999999999},"last":0.59399999999999997}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.42299999999999999},"min":{"1min":0.27100000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.371},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.439,"15min":0.45500000000000002},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67200000000000004,"5min":0.69199999999999995,"15min":0.69199999999999995},"last":0.40899999999999997}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42799999999999999,"15min":0.46700000000000003},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65600000000000003,"5min":0.876,"15min":0.876},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.45700000000000002,"15min":0.46999999999999997},"min":{"1min":0.32800000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.65400000000000003,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.39100000000000001}]}]},{"osd":2,"up_from":11,"seq":47244640326,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35499999999999998,"15min":0.34999999999999998},"min":{"1min":0.20399999999999999,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.52300000000000002,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42399999999999999,"15min":0.44},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57999999999999996,"5min":1.298,"15min":1.298},"last":0.40400000000000003}]},{"osd":1,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.42299999999999999,"15min":0.435},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.56299999999999994,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.375,"5min":0.39000000000000001,"15min":0.40500000000000003},"min":{"1min":0.25,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59599999999999997,"5min":1.5,"15min":1.5},"last":0.433}]},{"osd":3,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.55700000000000005,"15min":0.57399999999999995},"min":{"1min":0.34000000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.73699999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57099999999999995,"15min":0.60499999999999998},"min":{"1min":0.33500000000000002,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.70699999999999996,"5min":1.343,"15min":1.343},"last":0.63500000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.58299999999999996,"15min":0.61299999999999999},"min":{"1min":0.38100000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.71199999999999997,"5min":1.4790000000000001,"15min":1.4790000000000001},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56999999999999995,"15min":0.59799999999999998},"min":{"1min":0.41299999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.746,"5min":1.4279999999999999,"15min":1.4279999999999999},"last":0.66100000000000003}]},{"osd":5,"last update":"Sat Sep 30 00:49:29 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.61099999999999999},"min":{"1min":0.36599999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.78200000000000003,"5min":1.5509999999999999,"15min":1.5509999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.57499999999999996,"15min":0.58999999999999997},"min":{"1min":0.376,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.77500000000000002,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.64400000000000002}]}]},{"osd":0,"up_from":11,"seq":47244640325,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5144,"kb_used_data":144,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":94366696,"statfs":{"total":96636764160,"available":96631496704,"internally_reserved":0,"allocated":147456,"data_stored":44873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.35399999999999998,"15min":0.37},"min":{"1min":0.26400000000000001,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.505,"5min":0.82999999999999996,"15min":0.82999999999999996},"last":0.374},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38300000000000001,"15min":0.39500000000000002},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.50700000000000001,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.437}]},{"osd":2,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.377,"15min":0.372},"min":{"1min":0.30199999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.30499999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.45800000000000002}]},{"osd":3,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54600000000000004,"15min":0.54400000000000004},"min":{"1min":0.435,"5min":0.37,"15min":0.37},"max":{"1min":0.65300000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.54700000000000004,"15min":0.55100000000000005},"min":{"1min":0.38100000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.68600000000000005}]},{"osd":4,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.61399999999999999,"15min":0.61799999999999999},"min":{"1min":0.42799999999999999,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.752,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.627,"5min":0.623,"15min":0.629},"min":{"1min":0.44500000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.76500000000000001,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.77400000000000002}]},{"osd":5,"last update":"Sat Sep 30 00:49:33 2023","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.39100000000000001,"5min":0.372,"15min":0.372},"max":{"1min":0.71799999999999997,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.433,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.73099999999999998,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.70399999999999996}]}]},{"osd":1,"up_from":11,"seq":47244640325,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":5652,"kb_used_data":204,"kb_used_omap":0,"kb_used_meta":5440,"kb_avail":94366188,"statfs":{"total":96636764160,"available":96630976512,"internally_reserved":0,"allocated":208896,"data_stored":573801,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":5570560},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.34599999999999997,"15min":0.34200000000000003},"min":{"1min":0.28100000000000003,"5min":0.251,"15min":0.251},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.37,"15min":0.372},"min":{"1min":0.30099999999999999,"5min":0.248,"15min":0.248},"max":{"1min":0.52400000000000002,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.52500000000000002}]},{"osd":2,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.39600000000000002,"15min":0.38700000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.443,"5min":0.41399999999999998,"15min":0.41699999999999998},"min":{"1min":0.29799999999999999,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.58599999999999997}]},{"osd":3,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54100000000000004,"15min":0.55400000000000005},"min":{"1min":0.41199999999999998,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.79000000000000004,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.60299999999999998,"5min":0.56499999999999995,"15min":0.55400000000000005},"min":{"1min":0.44900000000000001,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.77000000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.63900000000000001}]},{"osd":4,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.628,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77300000000000002,"5min":0.84099999999999997,"15min":0.84099999999999997},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.621,"5min":0.61599999999999999,"15min":0.626},"min":{"1min":0.45400000000000001,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.78200000000000003,"5min":0.92600000000000005,"15min":0.92600000000000005},"last":0.88200000000000001}]},{"osd":5,"last update":"Sat Sep 30 00:49:35 2023","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.56799999999999995,"15min":0.56899999999999995},"min":{"1min":0.40500000000000003,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.745,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.89600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.59099999999999997},"min":{"1min":0.40400000000000003,"5min":0.377,"15min":0.377},"max":{"1min":0.78000000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.84299999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":29583,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-09-30T00:50:04.977 DEBUG:teuthology.orchestra.run.smithi006:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.107+0000 7ff49bed5700 10 monclient: get_monmap_and_config 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.107+0000 7ff49bed5700 10 monclient: build_initial_monmap 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.107+0000 7ff49bed5700 10 monclient: monmap: 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:50:05.108419+0000 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:50:05.108419+0000 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:05.108 INFO:teuthology.orchestra.run.smithi006.stderr:0: v1:172.21.15.6:6789/0 mon.noname-a-legacy 2023-09-30T00:50:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:1: v1:172.21.15.22:6789/0 mon.noname-c-legacy 2023-09-30T00:50:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-b 2023-09-30T00:50:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:3: v2:172.21.15.6:3300/0 mon.noname-a 2023-09-30T00:50:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:4: v2:172.21.15.22:3300/0 mon.noname-c 2023-09-30T00:50:05.109 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:05.109 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.107+0000 7ff49bed5700 1 Processor -- start 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 1 -- start start 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient: init 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient: _add_conns ranks=[3,1,2,0,4] 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 0x7ff494132d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): picked mon.noname-a con 0x7ff494098670 addr v2:172.21.15.6:3300/0 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): picked mon.noname-c-legacy con 0x7ff494098a40 addr v1:172.21.15.22:6789/0 2023-09-30T00:50:05.113 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 0x7ff49412f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): picked mon.noname-b con 0x7ff494098e90 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 1 -- --> v1:172.21.15.22:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4940a1000 con 0x7ff494098a40 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 1 -- --> v2:172.21.15.6:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4940a0a50 con 0x7ff494098670 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940a0730 con 0x7ff494098e90 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49bed5700 10 monclient(hunting): authenticate will time out at 1601.406710s 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 0x7ff494132d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 0x7ff49412f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 1 --2- >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 0x7ff494132d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42988/0 (socket says 172.21.15.6:42988) 2023-09-30T00:50:05.114 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 1 -- 172.21.15.6:0/2525325134 learned_addr learned my addr 172.21.15.6:0/2525325134 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): get_auth_request con 0x7ff494098670 auth_method 0 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): get_auth_request con 0x7ff494098e90 auth_method 0 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff49a472700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:05.115 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.108+0000 7ff499c71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 10 monclient(hunting): handle_auth_done global_id 4868 payload 274 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 1 -- 172.21.15.6:0/2525325134 >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 msgr2=0x7ff494132d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 1 --2- 172.21.15.6:0/2525325134 >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 0x7ff494132d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 1 -- 172.21.15.6:0/2525325134 >> v1:172.21.15.22:6789/0 conn(0x7ff494098a40 legacy=0x7ff49412ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 10 monclient: _finish_hunting 0 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 1 monclient: found mon.noname-b 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 10 monclient: _send_mon_message to mon.noname-b at v2:172.21.15.6:3301/0 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff49a472700 1 -- 172.21.15.6:0/2525325134 --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff494068df0 con 0x7ff494098e90 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.109+0000 7ff499c71700 1 --2- 172.21.15.6:0/2525325134 >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 0x7ff494132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2023-09-30T00:50:05.116 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff49a472700 1 -- 172.21.15.6:0/2525325134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 msgr2=0x7ff49412f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff49a472700 1 --2- 172.21.15.6:0/2525325134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 0x7ff49412f4f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e60 tx=0x7ff48c009780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c001639e366db9c server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 1 -- 172.21.15.6:0/2525325134 <== mon.2 v2:172.21.15.6:3301/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00e070 con 0x7ff494098e90 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: dump: 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:05.117 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: _finish_auth 0 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff49bed5700 5 monclient: authenticate success, global_id 4868 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 1 -- 172.21.15.6:0/2525325134 <== mon.2 v2:172.21.15.6:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff48c00bd30 con 0x7ff494098e90 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff49bed5700 10 monclient: get_monmap_and_config success 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff49bed5700 4 set_mon_vals no callback set 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 1 -- 172.21.15.6:0/2525325134 <== mon.2 v2:172.21.15.6:3301/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff48c00bea0 con 0x7ff494098e90 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:05.118 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: got monmap 1 from mon.c (according to old e1) 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.111+0000 7ff498c6f700 10 monclient: dump: 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:05.119 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 10 monclient: shutdown 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 -- 172.21.15.6:0/2525325134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 msgr2=0x7ff49412f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 --2- 172.21.15.6:0/2525325134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 0x7ff49412f4f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7ff48c009e60 tx=0x7ff48c009780 comp rx=0 tx=0).stop 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 -- 172.21.15.6:0/2525325134 shutdown_connections 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 --2- 172.21.15.6:0/2525325134 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098e90 0x7ff49412f4f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 --2- 172.21.15.6:0/2525325134 >> v2:172.21.15.6:3300/0 conn(0x7ff494098670 0x7ff494132d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 -- 172.21.15.6:0/2525325134 >> 172.21.15.6:0/2525325134 conn(0x7ff4940a5480 msgr2=0x7ff4940a5880 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 -- 172.21.15.6:0/2525325134 shutdown_connections 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 1 -- 172.21.15.6:0/2525325134 wait complete. 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 10 monclient: build_initial_monmap 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.112+0000 7ff49bed5700 10 monclient: monmap: 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 0 2023-09-30T00:50:05.120 INFO:teuthology.orchestra.run.smithi006.stderr:fsid 00000000-0000-0000-0000-000000000000 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 0.000000 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:created 0.000000 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 0 (unknown) 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.noname-a 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.noname-b 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.noname-c 2023-09-30T00:50:05.121 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 Processor -- start 2023-09-30T00:50:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 -- start start 2023-09-30T00:50:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient: init 2023-09-30T00:50:05.125 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient: _reopen_session rank -1 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient: _add_conns ranks=[0,2,1] 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 0x7ff49412d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): picked mon.noname-a con 0x7ff494098670 addr [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098a40 0x7ff49412d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): picked mon.noname-c con 0x7ff494098a40 addr [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 --2- >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff494098e90 0x7ff49409d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): picked mon.noname-b con 0x7ff494098e90 addr [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 -- --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4940a1500 con 0x7ff494098670 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 -- --> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff49409d220 con 0x7ff494098e90 2023-09-30T00:50:05.126 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): start opening mon connection 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 1 -- --> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff49412ed40 con 0x7ff494098a40 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): _renew_subs 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.113+0000 7ff49bed5700 10 monclient(hunting): authenticate will time out at 1601.411913s 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098a40 0x7ff49412d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 0x7ff49412d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 1 --2- >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098a40 0x7ff49412d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3301/0 says I am v2:172.21.15.6:54776/0 (socket says 172.21.15.6:54776) 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 --2- >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 0x7ff49412d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.6:3300/0 says I am v2:172.21.15.6:42998/0 (socket says 172.21.15.6:42998) 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 1 -- 172.21.15.6:0/1399593517 learned_addr learned my addr 172.21.15.6:0/1399593517 (peer_addr_for_me v2:172.21.15.6:0/0) 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): get_auth_request con 0x7ff494098a40 auth_method 0 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:05.127 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): get_auth_request con 0x7ff494098670 auth_method 0 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): handle_auth_reply_more payload 9 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): handle_auth_reply_more payload_len 9 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499470700 10 monclient(hunting): handle_auth_reply_more responding with 36 bytes 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff49a472700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff494098e90 0x7ff49409d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:05.128 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff49a472700 10 monclient(hunting): get_auth_request con 0x7ff494098e90 auth_method 0 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff49a472700 10 monclient(hunting): get_auth_request method 2 preferred_modes [2,1] 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff49a472700 10 monclient(hunting): _init_auth method 2 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff49a472700 10 monclient(hunting): _init_auth creating new auth 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient(hunting): handle_auth_done global_id 4815 payload 824 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 -- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098a40 msgr2=0x7ff49412d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098a40 0x7ff49412d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 -- 172.21.15.6:0/1399593517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff494098e90 msgr2=0x7ff49409d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff494098e90 0x7ff49409d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient: _finish_hunting 0 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 monclient: found mon.noname-a 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 20 monclient: _un_backoff reopen_interval_multipler now 1 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 10 monclient: _send_mon_message to mon.noname-a at v2:172.21.15.6:3300/0 2023-09-30T00:50:05.129 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff499c71700 1 -- 172.21.15.6:0/1399593517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4900094f0 con 0x7ff494098670 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.114+0000 7ff49a472700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff494098e90 0x7ff49409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff499c71700 1 -- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 msgr2=0x7ff49412d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).operator() setting up a delay queue 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff499c71700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 0x7ff49412d2d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff490006fd0 tx=0x7ff490009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff49000ca60 con 0x7ff494098670 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: dump: 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.130 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: _finish_auth 0 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 20 monclient: _check_auth_rotating not needed by client.admin 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff49bed5700 5 monclient: authenticate success, global_id 4815 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff49000cbc0 con 0x7ff494098670 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: handle_config config(0 keys) v1 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff490013470 con 0x7ff494098670 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: handle_monmap mon_map magic: 0 v1 2023-09-30T00:50:05.131 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: got monmap 1 from mon.a (according to old e1) 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff489ffb700 10 monclient: dump: 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:epoch 1 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:fsid cb933378-6423-4c15-869d-4727cefb664a 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:last_changed 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:created 2023-09-30T00:43:44.286124+0000 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:min_mon_release 17 (quincy) 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:election_strategy: 1 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:0: [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] mon.a 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:1: [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] mon.b 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2: [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] mon.c 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr: 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff48bfff700 4 set_mon_vals no callback set 2023-09-30T00:50:05.132 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff49bed5700 10 monclient: _renew_subs 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff49bed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.116+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff48c009a20 con 0x7ff494098670 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.117+0000 7ff49bed5700 10 monclient: _renew_subs 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.117+0000 7ff49bed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.117+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff49412f400 con 0x7ff494098670 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 4 ==== mgrmap(e 5) v1 ==== 42506+0+0 (secure 0 0 0) 0x7ff490013860 con 0x7ff494098670 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff489ffb700 1 --2- 172.21.15.6:0/1399593517 >> v2:172.21.15.22:6812/142603 conn(0x7ff4780357a0 0x7ff478037c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff49bed5700 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff49bed5700 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff49bed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff49409cea0 con 0x7ff494098670 2023-09-30T00:50:05.133 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 3502+0+0 (secure 0 0 0) 0x7ff490047ca0 con 0x7ff494098670 2023-09-30T00:50:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff499470700 1 --2- 172.21.15.6:0/1399593517 >> v2:172.21.15.22:6812/142603 conn(0x7ff4780357a0 0x7ff478037c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2023-09-30T00:50:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.118+0000 7ff499470700 10 monclient: get_auth_request con 0x7ff4780357a0 auth_method 0 2023-09-30T00:50:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.121+0000 7ff499470700 1 --2- 172.21.15.6:0/1399593517 >> v2:172.21.15.22:6812/142603 conn(0x7ff4780357a0 0x7ff478037c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff49412e540 tx=0x7ff484007f60 comp rx=0 tx=0).ready entity=mgr.4102 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-09-30T00:50:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.123+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+173446 (secure 0 0 0) 0x7ff49409cea0 con 0x7ff494098670 2023-09-30T00:50:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.123+0000 7ff489ffb700 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2023-09-30T00:50:05.134 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.123+0000 7ff489ffb700 10 monclient: _finish_command 1 = system:0 2023-09-30T00:50:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.327+0000 7ff49bed5700 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T00:50:05.327 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.327+0000 7ff49bed5700 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T00:50:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.327+0000 7ff49bed5700 10 monclient: _send_mon_message to mon.a at v2:172.21.15.6:3300/0 2023-09-30T00:50:05.328 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.327+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 --> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7ff49409cea0 con 0x7ff494098670 2023-09-30T00:50:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.330+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7ff49001a530 con 0x7ff494098670 2023-09-30T00:50:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.330+0000 7ff489ffb700 10 monclient: handle_config config(1 keys) v1 2023-09-30T00:50:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.330+0000 7ff48bfff700 4 set_mon_vals no callback set 2023-09-30T00:50:05.331 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.330+0000 7ff48bfff700 10 set_mon_vals mon_health_to_clog = false 2023-09-30T00:50:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.331+0000 7ff489ffb700 1 -- 172.21.15.6:0/1399593517 <== mon.0 v2:172.21.15.6:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v2)=0 v2) v1 ==== 123+0+0 (secure 0 0 0) 0x7ff49409cea0 con 0x7ff494098670 2023-09-30T00:50:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.331+0000 7ff489ffb700 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2023-09-30T00:50:05.332 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.331+0000 7ff489ffb700 10 monclient: _finish_command 2 = system:0 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 >> v2:172.21.15.22:6812/142603 conn(0x7ff4780357a0 msgr2=0x7ff478037c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 --2- 172.21.15.6:0/1399593517 >> v2:172.21.15.22:6812/142603 conn(0x7ff4780357a0 0x7ff478037c50 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff49412e540 tx=0x7ff484007f60 comp rx=0 tx=0).stop 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 10 monclient: shutdown 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 20 monclient: shutdown discarding 0 pending message(s) 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 msgr2=0x7ff49412d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 0x7ff49412d2d0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7ff490006fd0 tx=0x7ff490009250 comp rx=0 tx=0).stop 2023-09-30T00:50:05.336 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 shutdown_connections 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 --2- 172.21.15.6:0/1399593517 >> v2:172.21.15.22:6812/142603 conn(0x7ff4780357a0 0x7ff478037c50 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.22:3300/0,v1:172.21.15.22:6789/0] conn(0x7ff494098e90 0x7ff49409d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3300/0,v1:172.21.15.6:6789/0] conn(0x7ff494098670 0x7ff49412d2d0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 --2- 172.21.15.6:0/1399593517 >> [v2:172.21.15.6:3301/0,v1:172.21.15.6:6790/0] conn(0x7ff494098a40 0x7ff49412d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 >> 172.21.15.6:0/1399593517 conn(0x7ff4940a5480 msgr2=0x7ff494069010 unknown :-1 s=STATE_NONE l=0).mark_down 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 shutdown_connections 2023-09-30T00:50:05.337 INFO:teuthology.orchestra.run.smithi006.stderr:2023-09-30T00:50:05.335+0000 7ff49bed5700 1 -- 172.21.15.6:0/1399593517 wait complete. 2023-09-30T00:50:05.349 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:50:05.351 INFO:teuthology.misc:Shutting down mds daemons... 2023-09-30T00:50:05.351 INFO:teuthology.misc:Shutting down osd daemons... 2023-09-30T00:50:05.351 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2023-09-30T00:50:05.351 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.384 INFO:tasks.ceph.osd.0:Stopped 2023-09-30T00:50:05.385 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2023-09-30T00:50:05.385 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.449 INFO:tasks.ceph.osd.1:Stopped 2023-09-30T00:50:05.450 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2023-09-30T00:50:05.450 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.518 INFO:tasks.ceph.osd.2:Stopped 2023-09-30T00:50:05.518 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2023-09-30T00:50:05.518 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.553 INFO:tasks.ceph.osd.3:Stopped 2023-09-30T00:50:05.553 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2023-09-30T00:50:05.553 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.623 INFO:tasks.ceph.osd.4:Stopped 2023-09-30T00:50:05.623 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2023-09-30T00:50:05.623 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.694 INFO:tasks.ceph.osd.5:Stopped 2023-09-30T00:50:05.694 INFO:teuthology.misc:Shutting down mgr daemons... 2023-09-30T00:50:05.694 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2023-09-30T00:50:05.694 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.762 INFO:tasks.ceph.mgr.x:Stopped 2023-09-30T00:50:05.762 INFO:teuthology.misc:Shutting down mon daemons... 2023-09-30T00:50:05.762 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2023-09-30T00:50:05.762 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.776 INFO:tasks.ceph.mon.a:Stopped 2023-09-30T00:50:05.776 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2023-09-30T00:50:05.776 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.830 INFO:tasks.ceph.mon.c:Stopped 2023-09-30T00:50:05.830 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2023-09-30T00:50:05.831 INFO:teuthology.orchestra.run:waiting for 300 2023-09-30T00:50:05.844 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2023-09-30T00:50:05.844 INFO:tasks.ceph.mon.b:Stopped 2023-09-30T00:50:05.844 INFO:tasks.ceph:Checking cluster log for badness... 2023-09-30T00:50:05.844 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-30T00:50:05.910 INFO:teuthology.orchestra.run.smithi006.stdout:1696034659.5723534 mgr.x (mgr.4102) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:50:05.911 WARNING:tasks.ceph:Found errors (ERR|WRN|SEC) in cluster log 2023-09-30T00:50:05.911 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-30T00:50:05.990 DEBUG:teuthology.orchestra.run.smithi006:> sudo egrep '\[ERR\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | head -n 1 2023-09-30T00:50:06.071 INFO:teuthology.orchestra.run.smithi006.stdout:1696034659.5723534 mgr.x (mgr.4102) 1 : cluster [ERR] Unhandled exception from module 'pg_autoscaler' while running on mgr.x: 'PgAutoscaler' object has no attribute 'noautoscale' 2023-09-30T00:50:06.071 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:50:06.071 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2023-09-30T00:50:06.273 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:50:06.273 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2023-09-30T00:50:06.356 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:50:06.356 DEBUG:teuthology.orchestra.run.smithi006:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2023-09-30T00:50:06.448 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:50:06.448 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2023-09-30T00:50:06.610 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:50:06.611 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2023-09-30T00:50:06.681 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:50:06.681 DEBUG:teuthology.orchestra.run.smithi022:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2023-09-30T00:50:06.757 INFO:tasks.ceph:Archiving mon data... 2023-09-30T00:50:06.759 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/data/mon.a.tgz 2023-09-30T00:50:06.759 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-09-30T00:50:06.782 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.GyHY7fgG9G 2023-09-30T00:50:06.782 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.GyHY7fgG9G 2023-09-30T00:50:06.885 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.GyHY7fgG9G 2023-09-30T00:50:06.985 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.GyHY7fgG9G is 97KB 2023-09-30T00:50:07.016 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.GyHY7fgG9G 2023-09-30T00:50:07.037 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/data/mon.c.tgz 2023-09-30T00:50:07.038 DEBUG:teuthology.orchestra.run.smithi006:> mktemp 2023-09-30T00:50:07.100 INFO:teuthology.orchestra.run.smithi006.stdout:/tmp/tmp.LrKplSbxcR 2023-09-30T00:50:07.100 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.LrKplSbxcR 2023-09-30T00:50:07.192 DEBUG:teuthology.orchestra.run.smithi006:> sudo chmod 0666 /tmp/tmp.LrKplSbxcR 2023-09-30T00:50:07.290 DEBUG:teuthology.orchestra.remote:smithi006:/tmp/tmp.LrKplSbxcR is 97KB 2023-09-30T00:50:07.320 DEBUG:teuthology.orchestra.run.smithi006:> rm -fr /tmp/tmp.LrKplSbxcR 2023-09-30T00:50:07.342 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/data/mon.b.tgz 2023-09-30T00:50:07.342 DEBUG:teuthology.orchestra.run.smithi022:> mktemp 2023-09-30T00:50:07.366 INFO:teuthology.orchestra.run.smithi022.stdout:/tmp/tmp.vXVehIFIzh 2023-09-30T00:50:07.366 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vXVehIFIzh 2023-09-30T00:50:07.463 DEBUG:teuthology.orchestra.run.smithi022:> sudo chmod 0666 /tmp/tmp.vXVehIFIzh 2023-09-30T00:50:07.563 DEBUG:teuthology.orchestra.remote:smithi022:/tmp/tmp.vXVehIFIzh is 97KB 2023-09-30T00:50:07.592 DEBUG:teuthology.orchestra.run.smithi022:> rm -fr /tmp/tmp.vXVehIFIzh 2023-09-30T00:50:07.614 INFO:tasks.ceph:Cleaning ceph cluster... 2023-09-30T00:50:07.614 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:50:07.616 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2023-09-30T00:50:07.687 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2023-09-30T00:50:07.688 DEBUG:teuthology.orchestra.run.smithi006:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:50:07.692 DEBUG:teuthology.orchestra.run.smithi022:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2023-09-30T00:50:07.726 INFO:teuthology.orchestra.run.smithi006.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:50:07.765 INFO:teuthology.orchestra.run.smithi022.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2023-09-30T00:50:07.772 INFO:tasks.ceph:Archiving crash dumps... 2023-09-30T00:50:07.774 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/remote/smithi006/crash 2023-09-30T00:50:07.775 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:50:07.810 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/remote/smithi022/crash 2023-09-30T00:50:07.811 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2023-09-30T00:50:07.850 INFO:tasks.ceph:Compressing logs... 2023-09-30T00:50:07.850 DEBUG:teuthology.orchestra.run.smithi006:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:50:07.854 DEBUG:teuthology.orchestra.run.smithi022:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:50:08.733 INFO:tasks.ceph:Archiving logs... 2023-09-30T00:50:08.733 DEBUG:teuthology.misc:Transferring archived files from smithi006:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/remote/smithi006/log 2023-09-30T00:50:08.734 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:50:08.987 DEBUG:teuthology.misc:Transferring archived files from smithi022:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/remote/smithi022/log 2023-09-30T00:50:08.988 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /var/log/ceph -- . 2023-09-30T00:50:09.094 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:50:09.156 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=dcb050d3fdfa4a1298644d493f223f62 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1895, in task healthy(ctx=ctx, config=dict(cluster=config['cluster'])) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph.py", line 1472, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_ceph_ceph-c_93dce5e17918f8aca65993a7a2dc40233a47b481/qa/tasks/ceph_manager.py", line 3189, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2023-09-30T00:50:09.159 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2023-09-30T00:50:09.180 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-09-30T00:50:09.195 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-09-30T00:50:09.195 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:50:09.215 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-09-30T00:50:09.298 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-09-30T00:50:09.306 INFO:teuthology.task.clock:Checking final clock skew... 2023-09-30T00:50:09.307 DEBUG:teuthology.orchestra.run.smithi006:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:50:09.310 DEBUG:teuthology.orchestra.run.smithi022:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-09-30T00:50:09.329 INFO:teuthology.orchestra.run.smithi006.stderr:bash: ntpq: command not found 2023-09-30T00:50:09.331 INFO:teuthology.orchestra.run.smithi022.stderr:bash: ntpq: command not found 2023-09-30T00:50:09.335 INFO:teuthology.orchestra.run.smithi006.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:50:09.335 INFO:teuthology.orchestra.run.smithi006.stdout:=============================================================================== 2023-09-30T00:50:09.335 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 50 -285us[ -279us] +/- 86ms 2023-09-30T00:50:09.335 INFO:teuthology.orchestra.run.smithi006.stdout:^* hv02.front.sepia.ceph.com 3 6 377 47 -352us[ -346us] +/- 52ms 2023-09-30T00:50:09.335 INFO:teuthology.orchestra.run.smithi006.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 +772us[ +778us] +/- 72ms 2023-09-30T00:50:09.336 INFO:teuthology.orchestra.run.smithi006.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:50:09.336 INFO:teuthology.orchestra.run.smithi022.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-09-30T00:50:09.337 INFO:teuthology.orchestra.run.smithi022.stdout:=============================================================================== 2023-09-30T00:50:09.337 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 48 -301us[ -287us] +/- 86ms 2023-09-30T00:50:09.337 INFO:teuthology.orchestra.run.smithi022.stdout:^* hv02.front.sepia.ceph.com 3 6 377 48 -356us[ -341us] +/- 52ms 2023-09-30T00:50:09.337 INFO:teuthology.orchestra.run.smithi022.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 50 +768us[ +782us] +/- 72ms 2023-09-30T00:50:09.337 INFO:teuthology.orchestra.run.smithi022.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-09-30T00:50:09.339 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-09-30T00:50:09.347 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-09-30T00:50:09.348 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-09-30T00:50:09.356 DEBUG:teuthology.orchestra.run.smithi006:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:50:09.487 DEBUG:teuthology.orchestra.run.smithi022:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-09-30T00:50:09.592 DEBUG:teuthology.orchestra.run.smithi006:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:50:09.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:50:09.631 DEBUG:teuthology.orchestra.run.smithi022:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-09-30T00:50:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:50:09.669 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-09-30T00:50:09.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-09-30T00:50:09.688 INFO:teuthology.task.internal:Duration was 1128.195880 seconds 2023-09-30T00:50:09.688 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-09-30T00:50:09.696 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-09-30T00:50:09.697 DEBUG:teuthology.orchestra.run.smithi006:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:50:09.699 DEBUG:teuthology.orchestra.run.smithi022:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-09-30T00:50:09.757 INFO:teuthology.orchestra.run.smithi006.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:50:09.769 INFO:teuthology.orchestra.run.smithi022.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-09-30T00:50:09.981 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-09-30T00:50:09.981 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi006.front.sepia.ceph.com 2023-09-30T00:50:09.982 DEBUG:teuthology.orchestra.run.smithi006:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:50:10.011 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi022.front.sepia.ceph.com 2023-09-30T00:50:10.011 DEBUG:teuthology.orchestra.run.smithi022:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-09-30T00:50:10.041 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-09-30T00:50:10.042 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:50:10.054 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-09-30T00:50:10.129 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-09-30T00:50:10.130 DEBUG:teuthology.orchestra.run.smithi006:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:50:10.132 DEBUG:teuthology.orchestra.run.smithi022:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-09-30T00:50:10.209 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-09-30T00:50:10.220 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-09-30T00:50:10.221 DEBUG:teuthology.orchestra.run.smithi006:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:50:10.253 DEBUG:teuthology.orchestra.run.smithi022:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-09-30T00:50:10.286 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-09-30T00:50:10.296 DEBUG:teuthology.orchestra.run.smithi006:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:50:10.299 DEBUG:teuthology.orchestra.run.smithi022:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:50:10.331 INFO:teuthology.orchestra.run.smithi006.stdout:kernel.core_pattern = core 2023-09-30T00:50:10.362 INFO:teuthology.orchestra.run.smithi022.stdout:kernel.core_pattern = core 2023-09-30T00:50:10.388 DEBUG:teuthology.orchestra.run.smithi006:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:50:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:50:10.424 DEBUG:teuthology.orchestra.run.smithi022:> test -e /home/ubuntu/cephtest/archive/coredump 2023-09-30T00:50:10.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-09-30T00:50:10.450 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-09-30T00:50:10.467 INFO:teuthology.task.internal:Transferring archived files... 2023-09-30T00:50:10.467 DEBUG:teuthology.misc:Transferring archived files from smithi006:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/remote/smithi006 2023-09-30T00:50:10.468 DEBUG:teuthology.orchestra.run.smithi006:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:50:10.514 DEBUG:teuthology.misc:Transferring archived files from smithi022:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719/remote/smithi022 2023-09-30T00:50:10.514 DEBUG:teuthology.orchestra.run.smithi022:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-09-30T00:50:10.558 INFO:teuthology.task.internal:Removing archive directory... 2023-09-30T00:50:10.559 DEBUG:teuthology.orchestra.run.smithi006:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:50:10.562 DEBUG:teuthology.orchestra.run.smithi022:> rm -rf -- /home/ubuntu/cephtest/archive 2023-09-30T00:50:10.616 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-09-30T00:50:10.629 INFO:teuthology.task.internal:Not uploading archives. 2023-09-30T00:50:10.629 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-09-30T00:50:10.639 INFO:teuthology.task.internal:Tidying up after the test... 2023-09-30T00:50:10.639 DEBUG:teuthology.orchestra.run.smithi006:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:50:10.642 DEBUG:teuthology.orchestra.run.smithi022:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-09-30T00:50:10.662 INFO:teuthology.orchestra.run.smithi006.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:50 /home/ubuntu/cephtest 2023-09-30T00:50:10.679 INFO:teuthology.orchestra.run.smithi022.stdout: 525356 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 30 00:50 /home/ubuntu/cephtest 2023-09-30T00:50:10.681 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-09-30T00:50:10.691 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-09-30T00:50:10.710 INFO:teuthology.nuke:Checking targets against current locks 2023-09-30T00:50:10.736 DEBUG:teuthology.nuke:shortname: smithi006 2023-09-30T00:50:10.736 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:50:10.759 DEBUG:teuthology.nuke:shortname: smithi022 2023-09-30T00:50:10.759 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-09-30T00:50:10.763 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi006.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:09:13.969016', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMnRvJR0L2T9BYkKpGQNJp0fSnvm/Iup0o098MrH+5RqggRpFysYNoYtSqH26RVshsFZM5mKCGbBP/pXIRjBSlo='} 2023-09-30T00:50:10.786 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi022.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-09-29_19:38:56-rados-wip-yuri4-testing-2023-09-29-0928-quincy-distro-default-smithi/7406719', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-09-30 00:09:13.967196', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIS9surE7teWMyjraGAsw5v/FL+wIMOQobe4oi+WNkyXfGPxNue/tZudQy9axQWyCS951ZOY870XlEUUAonQhwo='} 2023-09-30T00:50:10.819 INFO:teuthology.orchestra.console.smithi006:Power off 2023-09-30T00:50:10.819 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:50:10.845 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2023-09-30T00:50:10.846 DEBUG:teuthology.orchestra.console.smithi006:power off output: Chassis Power Control: Down/Off 2023-09-30T00:50:10.846 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:10.862 INFO:teuthology.orchestra.console.smithi022:Power off 2023-09-30T00:50:10.862 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-09-30T00:50:10.876 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-09-30T00:50:10.883 DEBUG:teuthology.orchestra.console.smithi022:power off output: Chassis Power Control: Down/Off 2023-09-30T00:50:10.883 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:10.907 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2023-09-30T00:50:14.878 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:14.908 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:14.986 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-09-30T00:50:15.015 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is on 2023-09-30T00:50:18.991 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:19.016 DEBUG:teuthology.orchestra.console.smithi022:pexpect command: ipmitool -H smithi022.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:19.101 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is on 2023-09-30T00:50:19.125 DEBUG:teuthology.orchestra.console.smithi022:check power output: Chassis Power is off 2023-09-30T00:50:19.226 INFO:teuthology.orchestra.console.smithi022:Power off completed 2023-09-30T00:50:23.105 DEBUG:teuthology.orchestra.console.smithi006:pexpect command: ipmitool -H smithi006.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-09-30T00:50:23.215 DEBUG:teuthology.orchestra.console.smithi006:check power output: Chassis Power is off 2023-09-30T00:50:23.316 INFO:teuthology.orchestra.console.smithi006:Power off completed 2023-09-30T00:50:23.420 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/mon-delay msgr/async-v2only objectstore/bluestore-comp-snappy rados supported-random-distro$/{centos_8} thrashers/sync-many workloads/pool-create-delete} duration: 1128.1958804130554 failure_reason: '"1696034659.5723534 mgr.x (mgr.4102) 1 : cluster [ERR] Unhandled exception from module ''pg_autoscaler'' while running on mgr.x: ''PgAutoscaler'' object has no attribute ''noautoscale''" in cluster log' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=dcb050d3fdfa4a1298644d493f223f62 status: fail success: false 2023-09-30T00:50:23.421 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-09-30T00:50:23.500 INFO:teuthology.run:FAIL